### Navigation

- index
- modules |
- next |
- previous |
- pandas 0.13.1 documentation »

# Intro to Data Structures

We’ll start with a quick, non-comprehensive overview of the fundamental data structures in pandas to get you started. The fundamental behavior about data types, indexing, and axis labeling / alignment apply across all of the objects. To get started, import numpy and load pandas into your namespace:

    In [1]: import numpy as np

    # will use a lot in examples
    In [2]: randn = np.random.randn

    In [3]: from pandas import *

Here is a basic tenet to keep in mind: data alignment is intrinsic. The link between labels and data will not be broken unless done so explicitly by you.

We’ll give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in separate sections.

When using pandas, we recommend the following import convention:

    import pandas as pd

## Series

Warning

In 0.13.0 `Series` has internaly been refactored to no longer sub-class `ndarray` but instead subclass `NDFrame`, similarly to the rest of the pandas containers. This should be a transparent change with only very limited API implications (See the Internal Refactoring)

`Series` is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is to call:

    >>> s = Series(data, index=index)

Here, `data` can be many different things:

> - a Python dict
> - an ndarray
> - a scalar value (like 5)

The passed index is a list of axis labels. Thus, this separates into a few cases depending on what data is:

From ndarray

If `data` is an ndarray, index must be the same length as data. If no index is passed, one will be created having values `[0, ..., len(data) - 1]`.

    In [4]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])

    In [5]: s
    Out[5]:
    a   -1.344
    b    0.845
    c    1.076
    d   -0.109
    e    1.644
    dtype: float64

    In [6]: s.index
    Out[6]: Index([u'a', u'b', u'c', u'd', u'e'], dtype='object')

    In [7]: Series(randn(5))
    Out[7]:
    0   -1.469
    1    0.357
    2   -0.675
    3   -1.777
    4   -0.969
    dtype: float64

Note

Starting in v0.8.0, pandas supports non-unique index values. If an operation that does not support duplicate index values is attempted, an exception will be raised at that time. The reason for being lazy is nearly all performance-based (there are many instances in computations, like parts of GroupBy, where the index is not used).

From dict

If `data` is a dict, if index is passed the values in data corresponding to the labels in the index will be pulled out. Otherwise, an index will be constructed from the sorted keys of the dict, if possible.

    In [8]: d = {'a' : 0., 'b' : 1., 'c' : 2.}

    In [9]: Series(d)
    Out[9]:
    a    0
    b    1
    c    2
    dtype: float64

    In [10]: Series(d, index=['b', 'c', 'd', 'a'])
    Out[10]:
    b     1
    c     2
    d   NaN
    a     0
    dtype: float64

Note

NaN (not a number) is the standard missing data marker used in pandas

From scalar value If `data` is a scalar value, an index must be provided. The value will be repeated to match the length of index

    In [11]: Series(5., index=['a', 'b', 'c', 'd', 'e'])
    Out[11]:
    a    5
    b    5
    c    5
    d    5
    e    5
    dtype: float64

### Series is ndarray-like

`Series` acts very similary to a `ndarray`, and is a valid argument to most NumPy functions. However, things like slicing also slice the index.

    In [12]: s[0]
    Out[12]: -1.3443118127316671

    In [13]: s[:3]
    Out[13]:
    a   -1.344
    b    0.845
    c    1.076
    dtype: float64

    In [14]: s[s > s.median()]
    Out[14]:
    c    1.076
    e    1.644
    dtype: float64

    In [15]: s[[4, 3, 1]]
    Out[15]:
    e    1.644
    d   -0.109
    b    0.845
    dtype: float64

    In [16]: np.exp(s)
    Out[16]:
    a    0.261
    b    2.328
    c    2.932
    d    0.897
    e    5.174
    dtype: float64

We will address array-based indexing in a separate section.

### Series is dict-like

A Series is like a fixed-size dict in that you can get and set values by index label:

    In [17]: s['a']
    Out[17]: -1.3443118127316671

    In [18]: s['e'] = 12.

    In [19]: s
    Out[19]:
    a    -1.344
    b     0.845
    c     1.076
    d    -0.109
    e    12.000
    dtype: float64

    In [20]: 'e' in s
    Out[20]: True

    In [21]: 'f' in s
    Out[21]: False

If a label is not contained, an exception is raised:

    >>> s['f']
    KeyError: 'f'

Using the `get` method, a missing label will return None or specified default:

    In [22]: s.get('f')

    In [23]: s.get('f', np.nan)
    Out[23]: nan

See also the section on attribute access.

### Vectorized operations and label alignment with Series

When doing data analysis, as with raw NumPy arrays looping through Series value-by-value is usually not necessary. Series can be also be passed into most NumPy methods expecting an ndarray.

    In [24]: s + s
    Out[24]:
    a    -2.689
    b     1.690
    c     2.152
    d    -0.218
    e    24.000
    dtype: float64

    In [25]: s * 2
    Out[25]:
    a    -2.689
    b     1.690
    c     2.152
    d    -0.218
    e    24.000
    dtype: float64

    In [26]: np.exp(s)
    Out[26]:
    a         0.261
    b         2.328
    c         2.932
    d         0.897
    e    162754.791
    dtype: float64

A key difference between Series and ndarray is that operations between Series automatically align the data based on label. Thus, you can write computations without giving consideration to whether the Series involved have the same labels.

    In [27]: s[1:] + s[:-1]
    Out[27]:
    a      NaN
    b    1.690
    c    2.152
    d   -0.218
    e      NaN
    dtype: float64

The result of an operation between unaligned Series will have the union of the indexes involved. If a label is not found in one Series or the other, the result will be marked as missing `NaN`. Being able to write code without doing any explicit data alignment grants immense freedom and flexibility in interactive data analysis and research. The integrated data alignment features of the pandas data structures set pandas apart from the majority of related tools for working with labeled data.

Note

In general, we chose to make the default result of operations between differently indexed objects yield the union of the indexes in order to avoid loss of information. Having an index label, though the data is missing, is typically important information as part of a computation. You of course have the option of dropping labels with missing data via the dropna function.

### Name attribute

Series can also have a `name` attribute:

    In [28]: s = Series(np.random.randn(5), name='something')

    In [29]: s
    Out[29]:
    0   -1.295
    1    0.414
    2    0.277
    3   -0.472
    4   -0.014
    Name: something, dtype: float64

    In [30]: s.name
    Out[30]: 'something'

The Series `name` will be assigned automatically in many cases, in particular when taking 1D slices of DataFrame as you will see below.

## DataFrame

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:

> - Dict of 1D ndarrays, lists, dicts, or Series
> - 2-D numpy.ndarray
> - Structured or record ndarray
> - A `Series`
> - Another `DataFrame`

Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.

If axis labels are not passed, they will be constructed from the input data based on common sense rules.

### From dict of Series or dicts

The result index will be the union of the indexes of the various Series. If there are any nested dicts, these will be first converted to Series. If no columns are passed, the columns will be the sorted list of dict keys.

    In [31]: d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']),
       ....:      'two' : Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
       ....:

    In [32]: df = DataFrame(d)

    In [33]: df
    Out[33]:
       one  two
    a    1    1
    b    2    2
    c    3    3
    d  NaN    4

    [4 rows x 2 columns]

    In [34]: DataFrame(d, index=['d', 'b', 'a'])
    Out[34]:
       one  two
    d  NaN    4
    b    2    2
    a    1    1

    [3 rows x 2 columns]

    In [35]: DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])
    Out[35]:
       two three
    d    4   NaN
    b    2   NaN
    a    1   NaN

    [3 rows x 2 columns]

The row and column labels can be accessed respectively by accessing the index and columns attributes:

Note

When a particular set of columns is passed along with a dict of data, the passed columns override the keys in the dict.

    In [36]: df.index
    Out[36]: Index([u'a', u'b', u'c', u'd'], dtype='object')

    In [37]: df.columns
    Out[37]: Index([u'one', u'two'], dtype='object')

### From dict of ndarrays / lists

The ndarrays must all be the same length. If an index is passed, it must clearly also be the same length as the arrays. If no index is passed, the result will be `range(n)`, where `n` is the array length.

    In [38]: d = {'one' : [1., 2., 3., 4.],
       ....:      'two' : [4., 3., 2., 1.]}
       ....:

    In [39]: DataFrame(d)
    Out[39]:
       one  two
    0    1    4
    1    2    3
    2    3    2
    3    4    1

    [4 rows x 2 columns]

    In [40]: DataFrame(d, index=['a', 'b', 'c', 'd'])
    Out[40]:
       one  two
    a    1    4
    b    2    3
    c    3    2
    d    4    1

    [4 rows x 2 columns]

### From structured or record array

This case is handled identically to a dict of arrays.

    In [41]: data = np.zeros((2,),dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])

    In [42]: data[:] = [(1,2.,'Hello'),(2,3.,"World")]

    In [43]: DataFrame(data)
    Out[43]:
       A  B      C
    0  1  2  Hello
    1  2  3  World

    [2 rows x 3 columns]

    In [44]: DataFrame(data, index=['first', 'second'])
    Out[44]:
            A  B      C
    first   1  2  Hello
    second  2  3  World

    [2 rows x 3 columns]

    In [45]: DataFrame(data, columns=['C', 'A', 'B'])
    Out[45]:
           C  A  B
    0  Hello  1  2
    1  World  2  3

    [2 rows x 3 columns]

Note

DataFrame is not intended to work exactly like a 2-dimensional NumPy ndarray.

### From a list of dicts

    In [46]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]

    In [47]: DataFrame(data2)
    Out[47]:
       a   b   c
    0  1   2 NaN
    1  5  10  20

    [2 rows x 3 columns]

    In [48]: DataFrame(data2, index=['first', 'second'])
    Out[48]:
            a   b   c
    first   1   2 NaN
    second  5  10  20

    [2 rows x 3 columns]

    In [49]: DataFrame(data2, columns=['a', 'b'])
    Out[49]:
       a   b
    0  1   2
    1  5  10

    [2 rows x 2 columns]

### From a Series

The result will be a DataFrame with the same index as the input Series, and with one column whose name is the original name of the Series (only if no other column name provided).

Missing Data

Much more will be said on this topic in the Missing data section. To construct a DataFrame with missing data, use `np.nan` for those values which are missing. Alternatively, you may pass a `numpy.MaskedArray` as the data argument to the DataFrame constructor, and its masked entries will be considered missing.

### Alternate Constructors

DataFrame.from_dict

`DataFrame.from_dict` takes a dict of dicts or a dict of array-like sequences and returns a DataFrame. It operates like the `DataFrame` constructor except for the `orient` parameter which is `'columns'` by default, but which can be set to `'index'` in order to use the dict keys as row labels.

DataFrame.from_records

`DataFrame.from_records` takes a list of tuples or an ndarray with structured dtype. Works analogously to the normal `DataFrame` constructor, except that index maybe be a specific field of the structured dtype to use as the index. For example:

    In [50]: data
    Out[50]:
    array([(1, 2.0, 'Hello'), (2, 3.0, 'World')],
          dtype=[('A', '<i4'), ('B', '<f4'), ('C', 'S10')])

    In [51]: DataFrame.from_records(data, index='C')
    Out[51]:
           A  B
    C
    Hello  1  2
    World  2  3

    [2 rows x 2 columns]

DataFrame.from_items

`DataFrame.from_items` works analogously to the form of the `dict` constructor that takes a sequence of `(key, value)` pairs, where the keys are column (or row, in the case of `orient='index'`) names, and the value are the column values (or row values). This can be useful for constructing a DataFrame with the columns in a particular order without having to pass an explicit list of columns:

    In [52]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])])
    Out[52]:
       A  B
    0  1  4
    1  2  5
    2  3  6

    [3 rows x 2 columns]

If you pass `orient='index'`, the keys will be the row labels. But in this case you must also pass the desired column names:

    In [53]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])],
       ....:                      orient='index', columns=['one', 'two', 'three'])
       ....:
    Out[53]:
       one  two  three
    A    1    2      3
    B    4    5      6

    [2 rows x 3 columns]

### Column selection, addition, deletion

You can treat a DataFrame semantically like a dict of like-indexed Series objects. Getting, setting, and deleting columns works with the same syntax as the analogous dict operations:

    In [54]: df['one']
    Out[54]:
    a     1
    b     2
    c     3
    d   NaN
    Name: one, dtype: float64

    In [55]: df['three'] = df['one'] * df['two']

    In [56]: df['flag'] = df['one'] > 2

    In [57]: df
    Out[57]:
       one  two  three   flag
    a    1    1      1  False
    b    2    2      4  False
    c    3    3      9   True
    d  NaN    4    NaN  False

    [4 rows x 4 columns]

Columns can be deleted or popped like with a dict:

    In [58]: del df['two']

    In [59]: three = df.pop('three')

    In [60]: df
    Out[60]:
       one   flag
    a    1  False
    b    2  False
    c    3   True
    d  NaN  False

    [4 rows x 2 columns]

When inserting a scalar value, it will naturally be propagated to fill the column:

    In [61]: df['foo'] = 'bar'

    In [62]: df
    Out[62]:
       one   flag  foo
    a    1  False  bar
    b    2  False  bar
    c    3   True  bar
    d  NaN  False  bar

    [4 rows x 3 columns]

When inserting a Series that does not have the same index as the DataFrame, it will be conformed to the DataFrame’s index:

    In [63]: df['one_trunc'] = df['one'][:2]

    In [64]: df
    Out[64]:
       one   flag  foo  one_trunc
    a    1  False  bar          1
    b    2  False  bar          2
    c    3   True  bar        NaN
    d  NaN  False  bar        NaN

    [4 rows x 4 columns]

You can insert raw ndarrays but their length must match the length of the DataFrame’s index.

By default, columns get inserted at the end. The `insert` function is available to insert at a particular location in the columns:

    In [65]: df.insert(1, 'bar', df['one'])

    In [66]: df
    Out[66]:
       one  bar   flag  foo  one_trunc
    a    1    1  False  bar          1
    b    2    2  False  bar          2
    c    3    3   True  bar        NaN
    d  NaN  NaN  False  bar        NaN

    [4 rows x 5 columns]

### Indexing / Selection

The basics of indexing are as follows:

Operation | Syntax | Result
--- | --- | ---
Select column | `df[col]` | Series
Select row by label | `df.loc[label]` | Series
Select row by integer location | `df.iloc[loc]` | Series
Slice rows | `df[5:10]` | DataFrame
Select rows by boolean vector | `df[bool_vec]` | DataFrame

Row selection, for example, returns a Series whose index is the columns of the DataFrame:

    In [67]: df.loc['b']
    Out[67]:
    one              2
    bar              2
    flag         False
    foo            bar
    one_trunc        2
    Name: b, dtype: object

    In [68]: df.iloc[2]
    Out[68]:
    one             3
    bar             3
    flag         True
    foo           bar
    one_trunc     NaN
    Name: c, dtype: object

For a more exhaustive treatment of more sophisticated label-based indexing and slicing, see the section on indexing. We will address the fundamentals of reindexing / conforming to new sets of lables in the section on reindexing.

### Data alignment and arithmetic

Data alignment between DataFrame objects automatically align on both the columns and the index (row labels). Again, the resulting object will have the union of the column and row labels.

    In [69]: df = DataFrame(randn(10, 4), columns=['A', 'B', 'C', 'D'])

    In [70]: df2 = DataFrame(randn(7, 3), columns=['A', 'B', 'C'])

    In [71]: df + df2
    Out[71]:
           A      B      C   D
    0 -1.473 -0.626 -0.773 NaN
    1  0.073 -0.519  2.742 NaN
    2  1.744 -1.325  0.075 NaN
    3 -1.366 -1.238 -1.782 NaN
    4  0.275 -0.613 -2.263 NaN
    5  1.263  2.338  1.260 NaN
    6 -1.216  3.371 -1.992 NaN
    7    NaN    NaN    NaN NaN
    8    NaN    NaN    NaN NaN
    9    NaN    NaN    NaN NaN

    [10 rows x 4 columns]

When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the DataFrame columns, thus broadcasting row-wise. For example:

    In [72]: df - df.iloc[0]
    Out[72]:
           A      B      C      D
    0  0.000  0.000  0.000  0.000
    1  1.168 -1.200  3.489  0.536
    2  1.703 -1.164  0.697 -0.485
    3  1.176  0.138  0.096 -0.972
    4 -0.825  1.136 -0.514 -2.309
    5  1.970  1.030  1.493 -0.020
    6 -1.849  0.981 -1.084 -1.306
    7  0.284  0.552 -0.296 -2.123
    8  1.132 -1.275  0.195 -1.017
    9  0.265  0.702  1.265  0.064

    [10 rows x 4 columns]

In the special case of working with time series data, if the Series is a TimeSeries (which it will be automatically if the index contains datetime objects), and the DataFrame index also contains dates, the broadcasting will be column-wise:

    In [73]: index = date_range('1/1/2000', periods=8)

    In [74]: df = DataFrame(randn(8, 3), index=index, columns=list('ABC'))

    In [75]: df
    Out[75]:
                    A      B      C
    2000-01-01  3.357 -0.317 -1.236
    2000-01-02  0.896 -0.488 -0.082
    2000-01-03 -2.183  0.380  0.085
    2000-01-04  0.432  1.520 -0.494
    2000-01-05  0.600  0.274  0.133
    2000-01-06 -0.024  2.410  1.451
    2000-01-07  0.206 -0.252 -2.214
    2000-01-08  1.063  1.266  0.299

    [8 rows x 3 columns]

    In [76]: type(df['A'])
    Out[76]: pandas.core.series.Series

    In [77]: df - df['A']
    Out[77]:
                A      B      C
    2000-01-01  0 -3.675 -4.594
    2000-01-02  0 -1.384 -0.978
    2000-01-03  0  2.563  2.268
    2000-01-04  0  1.088 -0.926
    2000-01-05  0 -0.326 -0.467
    2000-01-06  0  2.434  1.474
    2000-01-07  0 -0.458 -2.420
    2000-01-08  0  0.203 -0.764

    [8 rows x 3 columns]

Warning

    df - df['A']

is now deprecated and will be removed in a future release. The preferred way to replicate this behavior is

    df.sub(df['A'], axis=0)

For explicit control over the matching and broadcasting behavior, see the section on flexible binary operations.

Operations with scalars are just as you would expect:

    In [78]: df * 5 + 2
    Out[78]:
                     A       B      C
    2000-01-01  18.787   0.413 -4.181
    2000-01-02   6.481  -0.438  1.589
    2000-01-03  -8.915   3.902  2.424
    2000-01-04   4.162   9.600 -0.468
    2000-01-05   5.001   3.371  2.664
    2000-01-06   1.882  14.051  9.253
    2000-01-07   3.030   0.740 -9.068
    2000-01-08   7.317   8.331  3.497

    [8 rows x 3 columns]

    In [79]: 1 / df
    Out[79]:
                     A      B       C
    2000-01-01   0.298 -3.150  -0.809
    2000-01-02   1.116 -2.051 -12.159
    2000-01-03  -0.458  2.629  11.786
    2000-01-04   2.313  0.658  -2.026
    2000-01-05   1.666  3.647   7.525
    2000-01-06 -42.215  0.415   0.689
    2000-01-07   4.853 -3.970  -0.452
    2000-01-08   0.940  0.790   3.340

    [8 rows x 3 columns]

    In [80]: df ** 4
    Out[80]:
                        A       B          C
    2000-01-01  1.271e+02   0.010  2.336e+00
    2000-01-02  6.450e-01   0.057  4.574e-05
    2000-01-03  2.271e+01   0.021  5.182e-05
    2000-01-04  3.495e-02   5.338  5.939e-02
    2000-01-05  1.298e-01   0.006  3.118e-04
    2000-01-06  3.149e-07  33.744  4.427e+00
    2000-01-07  1.803e-03   0.004  2.401e+01
    2000-01-08  1.278e+00   2.570  8.032e-03

    [8 rows x 3 columns]

Boolean operators work as well:

    In [81]: df1 = DataFrame({'a' : [1, 0, 1], 'b' : [0, 1, 1] }, dtype=bool)

    In [82]: df2 = DataFrame({'a' : [0, 1, 1], 'b' : [1, 1, 0] }, dtype=bool)

    In [83]: df1 & df2
    Out[83]:
           a      b
    0  False  False
    1  False   True
    2   True  False

    [3 rows x 2 columns]

    In [84]: df1 | df2
    Out[84]:
          a     b
    0  True  True
    1  True  True
    2  True  True

    [3 rows x 2 columns]

    In [85]: df1 ^ df2
    Out[85]:
           a      b
    0   True   True
    1   True  False
    2  False   True

    [3 rows x 2 columns]

    In [86]: -df1
    Out[86]:
           a      b
    0  False   True
    1   True  False
    2  False  False

    [3 rows x 2 columns]

### Transposing

To transpose, access the `T` attribute (also the `transpose` function), similar to an ndarray:

    # only show the first 5 rows
    In [87]: df[:5].T
    Out[87]:
       2000-01-01  2000-01-02  2000-01-03  2000-01-04  2000-01-05
    A       3.357       0.896      -2.183       0.432       0.600
    B      -0.317      -0.488       0.380       1.520       0.274
    C      -1.236      -0.082       0.085      -0.494       0.133

    [3 rows x 5 columns]

### DataFrame interoperability with NumPy functions

Elementwise NumPy ufuncs (log, exp, sqrt, ...) and various other NumPy functions can be used with no issues on DataFrame, assuming the data within are numeric:

    In [88]: np.exp(df)
    Out[88]:
                     A       B      C
    2000-01-01  28.715   0.728  0.290
    2000-01-02   2.450   0.614  0.921
    2000-01-03   0.113   1.463  1.089
    2000-01-04   1.541   4.572  0.610
    2000-01-05   1.822   1.316  1.142
    2000-01-06   0.977  11.136  4.265
    2000-01-07   1.229   0.777  0.109
    2000-01-08   2.896   3.547  1.349

    [8 rows x 3 columns]

    In [89]: np.asarray(df)
    Out[89]:
    array([[ 3.3574, -0.3174, -1.2363],
           [ 0.8962, -0.4876, -0.0822],
           [-2.1829,  0.3804,  0.0848],
           [ 0.4324,  1.52  , -0.4937],
           [ 0.6002,  0.2742,  0.1329],
           [-0.0237,  2.4102,  1.4505],
           [ 0.2061, -0.2519, -2.2136],
           [ 1.0633,  1.2661,  0.2994]])

The dot method on DataFrame implements matrix multiplication:

    In [90]: df.T.dot(df)
    Out[90]:
            A       B      C
    A  18.562  -0.274 -4.715
    B  -0.274  10.344  4.184
    C  -4.715   4.184  8.897

    [3 rows x 3 columns]

Similarly, the dot method on Series implements dot product:

    In [91]: s1 = Series(np.arange(5,10))

    In [92]: s1.dot(s1)
    Out[92]: 255

DataFrame is not intended to be a drop-in replacement for ndarray as its indexing semantics are quite different in places from a matrix.

### Console display

Very large DataFrames will be truncated to display them in the console. You can also get a summary using `info()`. (Here I am reading a CSV version of the baseball dataset from the plyr R package):

    In [93]: baseball = read_csv('data/baseball.csv')

    In [94]: print(baseball)
                  id  year  stint team  lg   g  ab  r
    88641  womacto01  2006      2  CHN  NL  19  50  6 ...
    88643  schilcu01  2006      1  BOS  AL  31   2  0 ...
    88645  myersmi01  2006      1  NYA  AL  62   0  0 ...
    88649  helliri01  2006      1  MIL  NL  20   3  0 ...
    88650  johnsra05  2006      1  NYA  AL  33   6  0 ...
                 ...   ...    ...  ... ... ... ... ..

    [100 rows x 22 columns]

    In [95]: baseball.info()
    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 100 entries, 88641 to 89534
    Data columns (total 22 columns):
    id       100 non-null object
    year     100 non-null int64
    stint    100 non-null int64
    team     100 non-null object
    lg       100 non-null object
    g        100 non-null int64
    ab       100 non-null int64
    r        100 non-null int64
    h        100 non-null int64
    X2b      100 non-null int64
    X3b      100 non-null int64
    hr       100 non-null int64
    rbi      100 non-null float64
    sb       100 non-null float64
    cs       100 non-null float64
    bb       100 non-null int64
    so       100 non-null float64
    ibb      100 non-null float64
    hbp      100 non-null float64
    sh       100 non-null float64
    sf       100 non-null float64
    gidp     100 non-null float64
    dtypes: float64(9), int64(10), object(3)

However, using `to_string` will return a string representation of the DataFrame in tabular form, though it won’t always fit the console width:

    In [96]: print(baseball.iloc[-20:, :12].to_string())
                  id  year  stint team  lg    g   ab   r    h  X2b  X3b  hr
    89474  finlest01  2007      1  COL  NL   43   94   9   17    3    0   1
    89480  embreal01  2007      1  OAK  AL    4    0   0    0    0    0   0
    89481  edmonji01  2007      1  SLN  NL  117  365  39   92   15    2  12
    89482  easleda01  2007      1  NYN  NL   76  193  24   54    6    0  10
    89489  delgaca01  2007      1  NYN  NL  139  538  71  139   30    0  24
    89493  cormirh01  2007      1  CIN  NL    6    0   0    0    0    0   0
    89494  coninje01  2007      2  NYN  NL   21   41   2    8    2    0   0
    89495  coninje01  2007      1  CIN  NL   80  215  23   57   11    1   6
    89497  clemero02  2007      1  NYA  AL    2    2   0    1    0    0   0
    89498  claytro01  2007      2  BOS  AL    8    6   1    0    0    0   0
    89499  claytro01  2007      1  TOR  AL   69  189  23   48   14    0   1
    89501  cirilje01  2007      2  ARI  NL   28   40   6    8    4    0   0
    89502  cirilje01  2007      1  MIN  AL   50  153  18   40    9    2   2
    89521  bondsba01  2007      1  SFN  NL  126  340  75   94   14    0  28
    89523  biggicr01  2007      1  HOU  NL  141  517  68  130   31    3  10
    89525  benitar01  2007      2  FLO  NL   34    0   0    0    0    0   0
    89526  benitar01  2007      1  SFN  NL   19    0   0    0    0    0   0
    89530  ausmubr01  2007      1  HOU  NL  117  349  38   82   16    3   3
    89533   aloumo01  2007      1  NYN  NL   87  328  51  112   19    1  13
    89534  alomasa02  2007      1  NYN  NL    8   22   1    3    1    0   0

New since 0.10.0, wide DataFrames will now be printed across multiple rows by default:

    In [97]: DataFrame(randn(3, 12))
    Out[97]:
             0         1         2         3         4         5         6   \
    0 -0.863838  0.408204 -1.048089 -0.025747 -0.988387  0.094055  1.262731
    1  0.369374 -0.034571 -2.484478 -0.281461  0.030711  0.109121  1.126203
    2 -1.071357  0.441153  2.353925  0.583787  0.221471 -0.744471  0.758527

             7         8         9         10        11
    0  1.289997  0.082423 -0.055758  0.536580 -0.489682
    1 -0.977349  1.474071 -0.064034 -1.282782  0.781836
    2  1.729689 -0.964980 -0.845696 -1.340896  1.846883

    [3 rows x 12 columns]

You can change how much to print on a single row by setting the `line_width` option:

    In [98]: set_option('line_width', 40) # default is 80

    In [99]: DataFrame(randn(3, 12))
    Out[99]:
             0         1         2   \
    0 -1.328865  1.682706 -1.717693
    1  0.306996 -0.028665  0.384316
    2 -1.137707 -0.891060 -0.693921

             3         4         5   \
    0  0.888782  0.228440  0.901805
    1  1.574159  1.588931  0.476720
    2  1.613616  0.464000  0.227371

             6         7         8   \
    0  1.171216  0.520260 -1.197071
    1  0.473424 -0.242861 -0.014805
    2 -0.496922  0.306389 -2.290613

             9         10        11
    0 -1.066969 -0.303421 -0.858447
    1 -0.284319  0.650776 -1.461665
    2 -1.134623 -1.561819 -0.260838

    [3 rows x 12 columns]

You can also disable this feature via the `expand_frame_repr` option. This will print the table in one block.

### DataFrame column attribute access and IPython completion

If a DataFrame column label is a valid Python variable name, the column can be accessed like attributes:

    In [100]: df = DataFrame({'foo1' : np.random.randn(5),
       .....:                 'foo2' : np.random.randn(5)})
       .....:

    In [101]: df
    Out[101]:
           foo1      foo2
    0  0.281957 -0.368204
    1  1.523962 -1.144073
    2 -0.902937  0.861209
    3  0.068159  0.800193
    4 -0.057873  0.782098

    [5 rows x 2 columns]

    In [102]: df.foo1
    Out[102]:
    0    0.281957
    1    1.523962
    2   -0.902937
    3    0.068159
    4   -0.057873
    Name: foo1, dtype: float64

The columns are also connected to the IPython completion mechanism so they can be tab-completed:

    In [5]: df.fo<TAB>
    df.foo1  df.foo2

## Panel

Panel is a somewhat less-used, but still important container for 3-dimensional data. The term panel data is derived from econometrics and is partially responsible for the name pandas: pan(el)-da(ta)-s. The names for the 3 axes are intended to give some semantic meaning to describing operations involving panel data and, in particular, econometric analysis of panel data. However, for the strict purposes of slicing and dicing a collection of DataFrame objects, you may find the axis names slightly arbitrary:

> - items: axis 0, each item corresponds to a DataFrame contained inside
> - major_axis: axis 1, it is the index (rows) of each of the DataFrames
> - minor_axis: axis 2, it is the columns of each of the DataFrames

Construction of Panels works about like you would expect:

### From 3D ndarray with optional axis labels

    In [103]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
       .....:            major_axis=date_range('1/1/2000', periods=5),
       .....:            minor_axis=['A', 'B', 'C', 'D'])
       .....:

    In [104]: wp
    Out[104]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
    Items axis: Item1 to Item2
    Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
    Minor_axis axis: A to D

### From dict of DataFrame objects

    In [105]: data = {'Item1' : DataFrame(randn(4, 3)),
       .....:         'Item2' : DataFrame(randn(4, 2))}
       .....:

    In [106]: Panel(data)
    Out[106]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)
    Items axis: Item1 to Item2
    Major_axis axis: 0 to 3
    Minor_axis axis: 0 to 2

Note that the values in the dict need only be convertible to DataFrame. Thus, they can be any of the other valid inputs to DataFrame as per above.

One helpful factory method is `Panel.from_dict`, which takes a dictionary of DataFrames as above, and the following named parameters:

Parameter | Default | Description
--- | --- | ---
intersect | `False` | drops elements whose indices do not align
orient | `items` | use `minor` to use DataFrames’ columns as panel items

For example, compare to the construction above:

    In [107]: Panel.from_dict(data, orient='minor')
    Out[107]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 3 (items) x 4 (major_axis) x 2 (minor_axis)
    Items axis: 0 to 2
    Major_axis axis: 0 to 3
    Minor_axis axis: Item1 to Item2

Orient is especially useful for mixed-type DataFrames. If you pass a dict of DataFrame objects with mixed-type columns, all of the data will get upcasted to `dtype=object` unless you pass `orient='minor'`:

    In [108]: df = DataFrame({'a': ['foo', 'bar', 'baz'],
       .....:                 'b': np.random.randn(3)})
       .....:

    In [109]: df
    Out[109]:
         a         b
    0  foo -1.035260
    1  bar -0.438229
    2  baz  0.503703

    [3 rows x 2 columns]

    In [110]: data = {'item1': df, 'item2': df}

    In [111]: panel = Panel.from_dict(data, orient='minor')

    In [112]: panel['a']
    Out[112]:
      item1 item2
    0   foo   foo
    1   bar   bar
    2   baz   baz

    [3 rows x 2 columns]

    In [113]: panel['b']
    Out[113]:
          item1     item2
    0 -1.035260 -1.035260
    1 -0.438229 -0.438229
    2  0.503703  0.503703

    [3 rows x 2 columns]

    In [114]: panel['b'].dtypes
    Out[114]:
    item1    float64
    item2    float64
    dtype: object

Note

Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected feature-wise. A number of methods and options available in DataFrame are not available in Panel. This will get worked on, of course, in future releases. And faster if you join me in working on the codebase.

### From DataFrame using `to_panel` method

This method was introduced in v0.7 to replace `LongPanel.to_long`, and converts a DataFrame with a two-level index to a Panel.

    In [115]: midx = MultiIndex(levels=[['one', 'two'], ['x','y']], labels=[[1,1,0,0],[1,0,1,0]])

    In [116]: df = DataFrame({'A' : [1, 2, 3, 4], 'B': [5, 6, 7, 8]}, index=midx)

    In [117]: df.to_panel()
    Out[117]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 2 (items) x 2 (major_axis) x 2 (minor_axis)
    Items axis: A to B
    Major_axis axis: one to two
    Minor_axis axis: x to y

### Item selection / addition / deletion

Similar to DataFrame functioning as a dict of Series, Panel is like a dict of DataFrames:

    In [118]: wp['Item1']
    Out[118]:
                       A         B         C         D
    2000-01-01 -1.069094 -1.099248  0.255269  0.009750
    2000-01-02  0.661084  0.379319 -0.008434  1.952541
    2000-01-03 -1.056652  0.533946 -1.226970  0.040403
    2000-01-04 -0.507516 -0.230096  0.394500 -1.934370
    2000-01-05 -1.652499  1.488753 -0.896484  0.576897

    [5 rows x 4 columns]

    In [119]: wp['Item3'] = wp['Item1'] / wp['Item2']

The API for insertion and deletion is the same as for DataFrame. And as with DataFrame, if the item is a valid python identifier, you can access it as an attribute and tab-complete it in IPython.

### Transposing

A Panel can be rearranged using its `transpose` method (which does not make a copy by default unless the data are heterogeneous):

    In [120]: wp.transpose(2, 0, 1)
    Out[120]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 4 (items) x 3 (major_axis) x 5 (minor_axis)
    Items axis: A to D
    Major_axis axis: Item1 to Item3
    Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00

### Indexing / Selection

Operation | Syntax | Result
--- | --- | ---
Select item | `wp[item]` | DataFrame
Get slice at major_axis label | `wp.major_xs(val)` | DataFrame
Get slice at minor_axis label | `wp.minor_xs(val)` | DataFrame

For example, using the earlier example data, we could do:

    In [121]: wp['Item1']
    Out[121]:
                       A         B         C         D
    2000-01-01 -1.069094 -1.099248  0.255269  0.009750
    2000-01-02  0.661084  0.379319 -0.008434  1.952541
    2000-01-03 -1.056652  0.533946 -1.226970  0.040403
    2000-01-04 -0.507516 -0.230096  0.394500 -1.934370
    2000-01-05 -1.652499  1.488753 -0.896484  0.576897

    [5 rows x 4 columns]

    In [122]: wp.major_xs(wp.major_axis[2])
    Out[122]:
          Item1     Item2     Item3
    A -1.056652  1.375020 -0.768463
    B  0.533946 -0.928797 -0.574879
    C -1.226970 -0.308853  3.972672
    D  0.040403 -0.681087 -0.059321

    [4 rows x 3 columns]

    In [123]: wp.minor_axis
    Out[123]: Index([u'A', u'B', u'C', u'D'], dtype='object')

    In [124]: wp.minor_xs('C')
    Out[124]:
                   Item1     Item2     Item3
    2000-01-01  0.255269  0.604603  0.422209
    2000-01-02 -0.008434  0.967661 -0.008715
    2000-01-03 -1.226970 -0.308853  3.972672
    2000-01-04  0.394500 -2.461467 -0.160270
    2000-01-05 -0.896484  1.771740 -0.505991

    [5 rows x 3 columns]

### Squeezing

Another way to change the dimensionality of an object is to `squeeze` a 1-len object, similar to `wp['Item1']`

    In [125]: wp.reindex(items=['Item1']).squeeze()
    Out[125]:
                       A         B         C         D
    2000-01-01 -1.069094 -1.099248  0.255269  0.009750
    2000-01-02  0.661084  0.379319 -0.008434  1.952541
    2000-01-03 -1.056652  0.533946 -1.226970  0.040403
    2000-01-04 -0.507516 -0.230096  0.394500 -1.934370
    2000-01-05 -1.652499  1.488753 -0.896484  0.576897

    [5 rows x 4 columns]

    In [126]: wp.reindex(items=['Item1'],minor=['B']).squeeze()
    Out[126]:
    2000-01-01   -1.099248
    2000-01-02    0.379319
    2000-01-03    0.533946
    2000-01-04   -0.230096
    2000-01-05    1.488753
    Freq: D, Name: B, dtype: float64

### Conversion to DataFrame

A Panel can be represented in 2D form as a hierarchically indexed DataFrame. See the section hierarchical indexing for more on this. To convert a Panel to a DataFrame, use the `to_frame` method:

    In [127]: panel = Panel(np.random.randn(3, 5, 4), items=['one', 'two', 'three'],
       .....:               major_axis=date_range('1/1/2000', periods=5),
       .....:               minor_axis=['a', 'b', 'c', 'd'])
       .....:

    In [128]: panel.to_frame()
    Out[128]:
                           one       two     three
    major      minor
    2000-01-01 a      0.413086 -0.033277 -1.132896
               b     -1.139050  0.281151 -2.006481
               c      0.660342 -1.298915  0.301016
               d      0.464794 -2.819487  0.059117
    2000-01-02 a     -0.309337 -0.851985  1.138469
               b     -0.649593 -1.106952 -2.400634
               c      0.683758 -0.937731 -0.280853
               d     -0.643834 -1.537770  0.025653
    2000-01-03 a      0.421287  0.555759 -1.386071
               b      1.032814 -2.277282  0.863937
               c     -1.290493 -0.390201  0.252462
               d      0.787872  1.207122  1.500571
    2000-01-04 a      1.515707  0.178690  1.053202
               b     -0.276487 -1.004168 -2.338595
               c     -0.223762 -1.377627 -0.374279
               d      1.397431  0.499281 -2.359958
    2000-01-05 a      1.503874 -1.405256 -1.157886
               b     -0.478905  0.162565 -0.551865
               c     -0.135950 -0.067785  1.592673
               d     -0.730327 -1.260006  1.559318

    [20 rows x 3 columns]

## Panel4D (Experimental)

`Panel4D` is a 4-Dimensional named container very much like a `Panel`, but having 4 named dimensions. It is intended as a test bed for more N-Dimensional named containers.

> - labels: axis 0, each item corresponds to a Panel contained inside
> - items: axis 1, each item corresponds to a DataFrame contained inside
> - major_axis: axis 2, it is the index (rows) of each of the DataFrames
> - minor_axis: axis 3, it is the columns of each of the DataFrames

`Panel4D` is a sub-class of `Panel`, so most methods that work on Panels are applicable to Panel4D. The following methods are disabled:

> - `join , to_frame , to_excel , to_sparse , groupby`

Construction of Panel4D works in a very similar manner to a `Panel`

### From 4D ndarray with optional axis labels

    In [129]: p4d = Panel4D(randn(2, 2, 5, 4),
       .....:            labels=['Label1','Label2'],
       .....:            items=['Item1', 'Item2'],
       .....:            major_axis=date_range('1/1/2000', periods=5),
       .....:            minor_axis=['A', 'B', 'C', 'D'])
       .....:

    In [130]: p4d
    Out[130]:
    <class 'pandas.core.panelnd.Panel4D'>
    Dimensions: 2 (labels) x 2 (items) x 5 (major_axis) x 4 (minor_axis)
    Labels axis: Label1 to Label2
    Items axis: Item1 to Item2
    Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
    Minor_axis axis: A to D

### From dict of Panel objects

    In [131]: data = { 'Label1' : Panel({ 'Item1' : DataFrame(randn(4, 3)) }),
       .....:          'Label2' : Panel({ 'Item2' : DataFrame(randn(4, 2)) }) }
       .....:

    In [132]: Panel4D(data)
    Out[132]:
    <class 'pandas.core.panelnd.Panel4D'>
    Dimensions: 2 (labels) x 2 (items) x 4 (major_axis) x 3 (minor_axis)
    Labels axis: Label1 to Label2
    Items axis: Item1 to Item2
    Major_axis axis: 0 to 3
    Minor_axis axis: 0 to 2

Note that the values in the dict need only be convertible to Panels. Thus, they can be any of the other valid inputs to Panel as per above.

### Slicing

Slicing works in a similar manner to a Panel. `[]` slices the first dimension. `.ix` allows you to slice abitrarily and get back lower dimensional objects

    In [133]: p4d['Label1']
    Out[133]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
    Items axis: Item1 to Item2
    Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
    Minor_axis axis: A to D

4D -> Panel

    In [134]: p4d.ix[:,:,:,'A']
    Out[134]:
    <class 'pandas.core.panel.Panel'>
    Dimensions: 2 (items) x 2 (major_axis) x 5 (minor_axis)
    Items axis: Label1 to Label2
    Major_axis axis: Item1 to Item2
    Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00

4D -> DataFrame

    In [135]: p4d.ix[:,:,0,'A']
    Out[135]:
             Label1    Label2
    Item1  1.562443  0.159653
    Item2 -0.015601  0.136235

    [2 rows x 2 columns]

4D -> Series

    In [136]: p4d.ix[:,0,0,'A']
    Out[136]:
    Label1    1.562443
    Label2    0.159653
    Name: A, dtype: float64

### Transposing

A Panel4D can be rearranged using its `transpose` method (which does not make a copy by default unless the data are heterogeneous):

    In [137]: p4d.transpose(3, 2, 1, 0)
    Out[137]:
    <class 'pandas.core.panelnd.Panel4D'>
    Dimensions: 4 (labels) x 5 (items) x 2 (major_axis) x 2 (minor_axis)
    Labels axis: A to D
    Items axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
    Major_axis axis: Item1 to Item2
    Minor_axis axis: Label1 to Label2

## PanelND (Experimental)

PanelND is a module with a set of factory functions to enable a user to construct N-dimensional named containers like Panel4D, with a custom set of axis labels. Thus a domain-specific container can easily be created.

The following creates a Panel5D. A new panel type object must be sliceable into a lower dimensional object. Here we slice to a Panel4D.

    In [138]: from pandas.core import panelnd

    In [139]: Panel5D = panelnd.create_nd_panel_factory(
       .....:     klass_name   = 'Panel5D',
       .....:     orders  = [ 'cool', 'labels','items','major_axis','minor_axis'],
       .....:     slices  = { 'labels' : 'labels', 'items' : 'items',
       .....:                     'major_axis' : 'major_axis', 'minor_axis' : 'minor_axis' },
       .....:     slicer  = Panel4D,
       .....:     aliases = { 'major' : 'major_axis', 'minor' : 'minor_axis' },
       .....:     stat_axis    = 2)
       .....:

    In [140]: p5d = Panel5D(dict(C1 = p4d))

    In [141]: p5d
    Out[141]:
    <class 'pandas.core.panelnd.Panel5D'>
    Dimensions: 1 (cool) x 2 (labels) x 2 (items) x 5 (major_axis) x 4 (minor_axis)
    Cool axis: C1 to C1
    Labels axis: Label1 to Label2
    Items axis: Item1 to Item2
    Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
    Minor_axis axis: A to D

    # print a slice of our 5D
    In [142]: p5d.ix['C1',:,:,0:3,:]
    Out[142]:
    <class 'pandas.core.panelnd.Panel4D'>
    Dimensions: 2 (labels) x 2 (items) x 3 (major_axis) x 4 (minor_axis)
    Labels axis: Label1 to Label2
    Items axis: Item1 to Item2
    Major_axis axis: 2000-01-01 00:00:00 to 2000-01-03 00:00:00
    Minor_axis axis: A to D

    # transpose it
    In [143]: p5d.transpose(1,2,3,4,0)
    Out[143]:
    <class 'pandas.core.panelnd.Panel5D'>
    Dimensions: 2 (cool) x 2 (labels) x 5 (items) x 4 (major_axis) x 1 (minor_axis)
    Cool axis: Label1 to Label2
    Labels axis: Item1 to Item2
    Items axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
    Major_axis axis: A to D
    Minor_axis axis: C1 to C1

    # look at the shape & dim
    In [144]: p5d.shape
    Out[144]: (1, 2, 2, 5, 4)

    In [145]: p5d.ndim
    Out[145]: 5

### Navigation

- index
- modules |
- next |
- previous |
- pandas 0.13.1 documentation »