Unlike common ordered states of matter such as the crystalline solids and magnets characterized by some broken symmetry, the topological states of quantum matter are characterized by boundary properties that are highly robust or stable in the presence of disorder, fluctuations, and perturbations [1, 2]. The first example of a two-dimensional topological insulator was provided by the quantum Hall effect in an electron gas [1], which has no distinct three-dimensional topological generalization [3,4, 5, 6]. Recent theoretical and experimental work indicates that strong spin-orbit coupling in insulators can produce a three-dimensional topological insulator, which is a completely new state of matter [3, 5, 6, 7,8, 9, 10, 11, 12, 13]. This state is characterized by a surface Berry’s phase, realized via an odd number of Dirac cones protected against disorder such as alloying [3, 4, 5, 6]. This is in sharp contrast to the case of graphene, which possesses an even number of Dirac cones.

The thermoelectric Bi-Sb alloys discovered last year provided the first example of a truly three-dimensional topological insulator [7, 8, 9]. The experimental discovery of another thermoelectric family followed—the Bi2X3 (X=Se,Te) series—harboring the same topological insulator states with a simpler surface spectrum [10, 11, 12, 13]. The large band gap topological insulator Bi2Se3 features a nearly ideal Dirac cone [10, 12], but in the smaller band gap material Bi2Te3, with trigonal crystal potential, the cone is warped [11, 12]. Even though the cone is deformed with hexagonal symmetry, the Berry phase quantifying the topological invariant remains unchanged, consistent with its topological order [12, 13]. Now, in an article appearing in the current issue of Physical Review Letters [14], Liang Fu of Harvard University in the US presents a perturbation calculation, which yields a hexagonal modulation of the surface states analogous to the existence of a nonlinear term in the spin-orbit interaction in the bulk crystal. (See Fig. 1 for an illustration using our results of a full-potential, self-consistent computation in Bi2Te3.) This leads to a number of remarkable predictions concerning the unusual physical properties induced by the warping of the surface states, which are distinct from the simpler “hydrogen-atom” version of the topological insulator discovered previously [10].

The spin and linear momentum of an electron on the surface of a topological insulator are locked-in one-to-one, as revealed in recent spin-resolved photoemission measurements [8, 9, 12, 13]: the electron is forbidden from moving backwards or taking a “U-turn”—an effect that hinders its localization [15]. This can be realized in a real material if the spins of electrons lying on the Dirac cone, but away from the apex (the Dirac node), rotate around in a circle at the Fermi level. Spins arranged in this way lead to a geometrical quantum phase known as Berry’s phase with a value of π [8, 12]. The work of Fu effectively shows that in the presence of a hexagonally deformed cone, the spins must acquire finite out-of-the-plane components to conserve the net value of Berry’s phase, thus preserving the bulk topological invariant. The resulting finite value of the out-of-the-plane component opens up new possibilities for observing spectacular quantum effects.

Fascinating classes of new particles such as a Majorana fermion, or a magnetic monopole image, or a massive spin-textured Dirac particle can live on the surface of a topological insulator if a finite gap can be induced in the surface band with the chemical potential near the Dirac node [3, 4, 5, 6, 16]. One approach is to create an interface between the topological insulator and a more conventional material such as a magnet or a superconductor, which can induce an energy gap in the Dirac bands. Alternatively, a magnetic field can be applied perpendicular to the sample surface. Until now it had been thought that a parallel magnetic field would neither open a gap nor lead to a surface Hall effect. However, when the surface states are warped significantly in a truly bulk insulating sample, as the work of Fu shows, a parallel magnetic field would open up a gap in the surface spectrum, enabling quantum Hall experiments to be carried out using relatively modest magnetic fields (less than 7 tesla) accessible in a typical laboratory setting. This is an exciting prospect since such experiments have so far not been possible because very high magnetic fields are needed for opening a gap.

In a truly bulk insulating sample, where the chemical potential is such that the hexagonal warping of the Fermi surface survives, one would expect to see a rare realization of the quantum Hall effect on the surface without the presence of Landau levels. In this way, one could observe a half-integer quantum of conductance per surface, which has not been possible so far in any known two-dimensional electron gas, including graphene, and thus allow experimental tests of currently debated scenarios for unusual antilocalization-to-localization transitions on a topological insulator [5]. Presently realized Bi2Te3 is quite bulk-metallic with small resistivity [11, 12], whereas spin-orbit insulators with high bulk resistivity values coupled with surface carrier control are needed to realize a functional topological insulator with which many more interesting experiments could be possible. Recent progress in making highly resistiveBi2Se3 [12, 17] coupled with surface carrier control [18] heralds the potential dawn of a topological revolution. Similar improvements in sample resistivity are also called for in Bi2Te3.

The hexagonal warping of the surface cone and the resulting deviations from a circular Fermi surface provide a natural explanation for the enhanced magnitude of surface electronic interference effects recently reported around crystalline defects in Bi2Te3 [19, 20]. For a circular Fermi surface (Fig. 1, left), the electronic states at the Fermi surface cannot backscatter since in a fully spin-polarized band [12, 13] a consequence of time-reversal symmetry is that there are no available states into which electrons can scatter elastically. As a result, interference effects around the defects are small. With hexagonal warping, however, new electronic scattering channels open up, connected by various simple nesting vectors on the Fermi surface (see Fig. 1), and consistent with recent scanning tunneling measurements, the scattering due to defects becomes enhanced [19, 20].

The ideal topological insulators displaying a perfect Dirac cone dispersion cannot support the formation of charge- or spin-density waves. However, in the presence of hexagonal or other forms of deformations of the surface band, such ubiquitous ordering instabilities are no longer forbidden, but rather present a competing ground state, which can circumvent the topological protection. Since electrons with opposite momenta possess opposite spins, a spin-density-wave state is favored if the Fermi surface has parallel segments that can be nested, as is seen to be the case for hexagonal deformation (see Fig 1, right). As one goes around the hexagonal Fermi surface, electrons will experience rapid changes in the out-of-the-plane spin component (σz) to maintain their overall Berry’s phase value at π [8, 12], making the resulting spin-density wave appear quite exotic. A stripelike one-dimensional order on the topological surface may also be energetically possible.

Even more complex spin-density-wave orderings could be supported by other topological insulators that presently lack a tractable theoretical model. For example, Bi1-xSbx alloys display a central hexagonally deformed Dirac cone and additional surface states of complex character [8]. Despite the complexity of its five surface bands and hexagonal warping, the Bi1-xSbx alloy topological insulator precisely exhibits π Berry’s phase and antilocalization just like the “hydrogen-atom” single-surface-Dirac-cone systems [8, 10, 12]. This is a remarkable manifestation and demonstration of a deep and profound principle of physics—topological invariance. Thus the power and beauty of the topological order and invariance in nature are rather firmly established by studying systems that have complex “geometries,” i.e., complex and warped surface states as in the Bi1-xSbx alloy series or Bi2Te3.

The most remarkable property of topological surface states is that they are expected to be perfect metals in the sense that they are robust and stable against most perturbations and instabilities towards conventional ordered states of matter. The warping of the surface states, even though it opens up interesting possibilities for studying this unusual state of matter in detail, ironically also opens up competing channels which provide a route for departure from the much sought after paradigm of perfect topological stability of the spin-textured metal [3, 4]. Most topological insulators lying hidden in nature are likely to harbor complex and warped surface states as in Bi1-xSbx and Bi2Te3.

Interestingly, even though the first three-dimensional topological insulators and their spin-textures carrying nontrivial Berry’s phases were discovered only recently, we already have new materials emerging with physical properties lying in a regime where quantum effects associated with competing orders such as the spin-density waves can be foreseen. Parallel progress in theoretical modeling, hand-in-hand with experimental advances in spin-sensitive measurement methods, will likely be essential for unraveling and eventually exploiting the exotic spin textures [21, 22] and other beautiful properties of topological quantum matter.