Fri Jul 16 14:43:23 IDT 2021
One of the essential pieces of NumPy is the ability to perform quick element-wise operations, both with basic arithmetic (addition, subtraction, multiplication, etc.) and with more sophisticated operations (trigonometric functions, exponential and logarithmic functions, etc.). Pandas inherits much of this functionality from NumPy, and the ufuncs that we introduced in Computation on NumPy Arrays: Universal Functions are key to this.
Dataset includes a couple useful twists, however: for unary operations like negation and trigonometric functions, these ufuncs will preserve index and column labels in the output, and for binary operations such as addition and multiplication, Pandas will automatically align indices when passing the objects to the ufunc. This means that keeping the context of data and combining data from different sources–both potentially error-prone tasks with raw NumPy arrays–become essentially foolproof ones with Pandas. We will additionally see that there are well-defined operations between one-dimensional Series structures and two-dimensional DataFrame structures.
(require
'[tech.v3.dataset :as ds]
'[tech.v3.datatype :as dtype]
'[tech.v3.datatype.functional :as dfn]
'[tablecloth.api :as tablecloth]
'[fastmath.random :as fm.rand])(def DS
(tablecloth/dataset
(zipmap [:A :B :C :D]
(repeatedly 4 (fn [] (repeatedly 3 #(fm.rand/frand 10)))))))^kind/dataset
DS_unnamed [3 4]:
| :A | :B | :C | :D |
|---|---|---|---|
| 3.09400129 | 4.33038664 | 8.30968666 | 4.57290792 |
| 7.32699347 | 8.63224316 | 0.67728257 | 7.74195385 |
| 7.00601959 | 7.86705065 | 8.69164276 | 9.59445572 |
If we apply a NumPy ufunc on either of these objects, the result will be another Pandas object with the indices preserved:
^kind/dataset
(ds/update-elemwise DS dfn/exp)_unnamed [3 4]:
| :A | :B | :C | :D |
|---|---|---|---|
| 22.06519087 | 75.97365524 | 4063.03967449 | 96.82526136 |
| 1520.80254620 | 5609.64746699 | 1.96852114 | 2302.96764301 |
| 1103.25435138 | 2609.85683869 | 5952.95349580 | 14683.14797648 |
Or, for a slightly more complex calculation:
^kind/dataset
(ds/update-elemwise DS #(dfn// (dfn/* % Math/PI)))_unnamed [3 4]:
| :A | :B | :C | :D |
|---|---|---|---|
| 0.10287969 | 0.07350611 | 0.03830588 | 0.06960776 |
| 0.04344345 | 0.03687453 | 0.46998092 | 0.04111493 |
| 0.04543377 | 0.04046115 | 0.03662252 | 0.03317644 |
Any of the ufuncs discussed in Computation on NumPy Arrays: Universal Functions can be used in a similar manner.
For binary operations on two Series or DataFrame objects, Pandas will align indices in the process of performing the operation. This is very convenient when working with incomplete data, as we'll see in some of the examples that follow.
A similar type of alignment takes place for both columns and indices when performing operations on DataFrames:
(def A
(tablecloth/dataset
(zipmap [:A :B]
(repeatedly 2 (fn [] (repeatedly 2 #(fm.rand/irand 20)))))))^kind/dataset
A_unnamed [2 2]:
| :A | :B |
|---|---|
| 6 | 15 |
| 8 | 13 |
(def B
(tablecloth/dataset
(zipmap [:B :A :C]
(repeatedly 3 (fn [] (repeatedly 3 #(fm.rand/irand 20)))))))^kind/dataset
B_unnamed [3 3]:
| :B | :A | :C |
|---|---|---|
| 5 | 2 | 1 |
| 11 | 19 | 10 |
| 19 | 17 | 7 |
Notice that indices are aligned correctly irrespective of their order in the two objects, and indices in the result are sorted. As was the case with Series, we can use the associated object's arithmetic method and pass any desired fill_value to be used in place of missing entries. Here we'll fill with the mean of all values in A (computed by first stacking the rows of A):
fill = A.stack().mean()
A.add(B, fill_value=fill)The following table lists Python operators and their equivalent Pandas object methods:
+ add()
- sub(), subtract()
* mul(), multiply()
/ truediv(), div(), divide()
// floordiv()
% mod()
** pow()When performing operations between a DataFrame and a Series, the index and column alignment is similarly maintained. Operations between a DataFrame and a Series are similar to operations between a two-dimensional and one-dimensional NumPy array. Consider one common operation, where we find the difference of a two-dimensional array and one of its rows: