The Fontaine-Mazur Conjecture for number fields predicts that infinite $\ell$-adic analytic groups cannot occur as the Galois groups of unramified $\ell$-extensions of number fields. We investigate the analogous question for function fields of one variable over finite fields, and then prove some special cases of both the number field and function field questions using ideas from class field theory, $\ell$-adic analytic groups, Lie algebras, arithmetic algebraic geometry, and Iwasawa theory.