towhee.types.tensor_array.TensorArray¶
- class towhee.types.tensor_array.TensorArray[source]¶
Bases:
ExtensionArray
Array for ndarrays
Methods
Return a list of Buffer objects pointing to this array's physical storage.
Cast array values to another data type
Get the chunks of the TensorArray.
Compute dictionary-encoded representation of array.
Compare contents of this array against another one.
Remove missing values from an array.
See
pyarrow.compute.fill_null()
for usage.Select values from an array.
Construct an Array from a sequence of buffers.
Create a TensroArray from numpy array.
Convert pandas.Series to an Arrow Array.
Construct ExtensionArray from type and storage array.
The sum of bytes in each buffer referenced by the array.
Find the first index of a value.
Return BooleanArray indicating the null values.
Return BooleanArray indicating the non-null values.
Compute zero-copy slice of this array.
Sum the values in a numerical array.
Select values from an array.
Create a numpy array from the TensorArray.
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
Convert to a list of native Python objects.
Render a "pretty-printed" string representation of the Array.
Alias of to_pylist for compatibility with NumPy.
Compute distinct elements in array.
Perform validation checks.
Compute counts of unique elements in array.
Return zero-copy "view" of array as another data type.
Attributes
Total number of bytes consumed by the elements of the array.
null_count
A relative position into another array's data.
storage
type
- __getitem__(index)[source]¶
Examples
>>> import numpy as np >>> from towhee.types import TensorArray >>> arr = TensorArray.from_numpy(np.arange(10).reshape([5,2])) >>> arr[0] array([0, 1])
- __init__(*args, **kwargs)¶
- __repr__()¶
Return repr(self).
- buffers(self)¶
Return a list of Buffer objects pointing to this array’s physical storage.
To correctly interpret these buffers, you need to also apply the offset multiplied with the size of the stored data type.
- cast(self, target_type=None, safe=None, options=None)¶
Cast array values to another data type
See
pyarrow.compute.cast()
for usage.- Parameters:
target_type (DataType, default None) – Type to cast array to.
safe (boolean, default True) – Whether to check for conversion errors such as overflow.
options (CastOptions, default None) – Additional checks pass by CastOptions
- Returns:
cast
- Return type:
- chunks(chunk_size=None)[source]¶
Get the chunks of the TensorArray.
Examples
>>> import numpy as np >>> from towhee.types import TensorArray >>> arr = TensorArray.from_numpy(np.arange(10).reshape([5,2])) >>> chunks = arr.chunks(2) >>> next(chunks) array([[0, 1], [2, 3]])
- dictionary_encode(self, null_encoding='mask')¶
Compute dictionary-encoded representation of array.
See
pyarrow.compute.dictionary_encode()
for full usage.- Parameters:
null_encoding – How to handle null entries.
- Returns:
encoded – A dictionary-encoded version of this array.
- Return type:
DictionaryArray
- diff(self, Array other)¶
Compare contents of this array against another one.
Return a string containing the result of diffing this array (on the left side) against the other array (on the right side).
- Parameters:
other (Array) – The other array to compare this array with.
- Returns:
diff – A human-readable printout of the differences.
- Return type:
str
Examples
>>> import pyarrow as pa >>> left = pa.array(["one", "two", "three"]) >>> right = pa.array(["two", None, "two-and-a-half", "three"]) >>> print(left.diff(right))
@@ -0, +0 @@ -“one” @@ -2, +1 @@ +null +”two-and-a-half”
- drop_null(self)¶
Remove missing values from an array.
- equals(self, Array other)¶
- fill_null(self, fill_value)¶
See
pyarrow.compute.fill_null()
for usage.- Parameters:
fill_value – The replacement value for null entries.
- Returns:
result – A new array with nulls replaced by the given value.
- Return type:
- filter(self, Array mask, *, null_selection_behavior=u'drop')¶
Select values from an array.
See
pyarrow.compute.filter()
for full usage.
- format(self, **kwargs)¶
- static from_buffers(DataType type, length, buffers, null_count=-1, offset=0, children=None)¶
Construct an Array from a sequence of buffers.
The concrete type returned depends on the datatype.
- Parameters:
type (DataType) – The value type of the array.
length (int) – The number of values in the array.
buffers (List[Buffer]) – The buffers backing this array.
null_count (int, default -1) – The number of null entries in the array. Negative value means that the null count is not known.
offset (int, default 0) – The array’s logical offset (in values, not in bytes) from the start of each buffer.
children (List[Array], default None) – Nested type children with length matching type.num_fields.
- Returns:
array
- Return type:
- classmethod from_numpy(data)[source]¶
Create a TensroArray from numpy array.
- Parameters:
data (numpy.ndarray) – The ndarray to create the TensorArray from.
Examples
>>> import numpy as np >>> from towhee.types import TensorArray >>> arr = TensorArray.from_numpy(np.arange(10).reshape([5,2])) >>> arr[0] array([0, 1])
>>> arr = TensorArray.from_numpy(np.arange(36).reshape([6,2,3])) >>> arr[1] array([[ 6, 7, 8], [ 9, 10, 11]]) >>> list(arr.chunks(5))[1] array([[[30, 31, 32], [33, 34, 35]]])
- static from_pandas(obj, mask=None, type=None, bool safe=True, MemoryPool memory_pool=None)¶
Convert pandas.Series to an Arrow Array.
This method uses Pandas semantics about what values indicate nulls. See pyarrow.array for more general conversion from arrays or sequences to Arrow arrays.
- Parameters:
obj (ndarray, pandas.Series, array-like) –
mask (array (boolean), optional) – Indicate which values are null (True) or not null (False).
type (pyarrow.DataType) – Explicit type to attempt to coerce to, otherwise will be inferred from the data.
safe (bool, default True) – Check for overflows or other unsafe conversions.
memory_pool (pyarrow.MemoryPool, optional) – If not passed, will allocate memory from the currently-set default memory pool.
Notes
Localized timestamps will currently be returned as UTC (pandas’s native representation). Timezone-naive data will be implicitly interpreted as UTC.
- Returns:
array – ChunkedArray is returned if object data overflows binary buffer.
- Return type:
pyarrow.Array or pyarrow.ChunkedArray
- static from_storage(BaseExtensionType typ, Array storage)¶
Construct ExtensionArray from type and storage array.
- Parameters:
typ (DataType) – The extension type for the result array.
storage (Array) – The underlying storage for the result array.
- Returns:
ext_array
- Return type:
ExtensionArray
- get_total_buffer_size(self)¶
The sum of bytes in each buffer referenced by the array.
An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.
If a buffer is referenced multiple times then it will only be counted once.
- index(self, value, start=None, end=None, *, memory_pool=None)¶
Find the first index of a value.
See
pyarrow.compute.index()
for full usage.- Parameters:
value (Scalar or object) – The value to look for in the array.
start (int, optional) – The start index where to look for value.
end (int, optional) – The end index where to look for value.
memory_pool (MemoryPool, optional) – A memory pool for potential memory allocations.
- Returns:
index – The index of the value in the array (-1 if not found).
- Return type:
Int64Scalar
- is_null(self, *, nan_is_null=False)¶
Return BooleanArray indicating the null values.
- Parameters:
nan_is_null (bool (optional, default False)) – Whether floating-point NaN values should also be considered null.
- Returns:
array
- Return type:
boolean Array
- is_valid(self)¶
Return BooleanArray indicating the non-null values.
- nbytes¶
Total number of bytes consumed by the elements of the array.
In other words, the sum of bytes from all buffer ranges referenced.
Unlike get_total_buffer_size this method will account for array offsets.
If buffers are shared between arrays then the shared portion will be counted multiple times.
The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.
- offset¶
A relative position into another array’s data.
The purpose is to enable zero-copy slicing. This value defaults to zero but must be applied on all operations with the physical storage buffers.
- slice(self, offset=0, length=None)¶
Compute zero-copy slice of this array.
- Parameters:
offset (int, default 0) – Offset from start of array to slice.
length (int, default None) – Length of slice (default is until end of Array starting from offset).
- Returns:
sliced
- Return type:
RecordBatch
- sum(self, **kwargs)¶
Sum the values in a numerical array.
See
pyarrow.compute.sum()
for full usage.- Parameters:
**kwargs (dict, optional) – Options to pass to
pyarrow.compute.sum()
.- Returns:
sum – A scalar containing the sum value.
- Return type:
Scalar
- take(self, indices)¶
Select values from an array.
See
pyarrow.compute.take()
for full usage.
- to_numpy(zero_copy_only=True)[source]¶
Create a numpy array from the TensorArray.
- Parameters:
zero_copy_only (bool) – Whether to create a copy of the array.
Examples
>>> import numpy as np >>> from towhee.types import TensorArray >>> arr = TensorArray.from_numpy(np.arange(10).reshape([5,2])) >>> arr.to_numpy() array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]])
- to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)¶
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
- Parameters:
memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.
strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.
categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.
integer_object_nulls (bool, default False) – Cast integers with nulls to objects
date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.
timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.
use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.
deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present
safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.
self_destruct (bool, default False) –
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.
Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.
types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or
None
if the default conversion should be used for that type. If you have a dictionary mapping, you can passdict.get
as function.
- Return type:
pandas.Series or pandas.DataFrame depending on type of object
Examples
>>> import pyarrow as pa >>> import pandas as pd
Convert a Table to pandas DataFrame:
>>> table = pa.table([ ... pa.array([2, 4, 5, 100]), ... pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) ... ], names=['n_legs', 'animals']) >>> table.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(table.to_pandas(), pd.DataFrame) True
Convert a RecordBatch to pandas DataFrame:
>>> import pyarrow as pa >>> n_legs = pa.array([2, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch pyarrow.RecordBatch n_legs: int64 animals: string >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(batch.to_pandas(), pd.DataFrame) True
Convert a Chunked Array to pandas Series:
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.to_pandas() 0 2 1 2 2 4 3 4 4 5 5 100 dtype: int64 >>> isinstance(n_legs.to_pandas(), pd.Series) True
- to_pylist(self)¶
Convert to a list of native Python objects.
- Returns:
lst
- Return type:
list
- to_string(self, *, int indent=2, int top_level_indent=0, int window=10, int container_window=2, bool skip_new_lines=False)¶
Render a “pretty-printed” string representation of the Array.
- Parameters:
indent (int, default 2) – How much to indent the internal items in the string to the right, by default
2
.top_level_indent (int, default 0) – How much to indent right the entire content of the array, by default
0
.window (int) – How many primitive items to preview at the begin and end of the array when the array is bigger than the window. The other items will be ellipsed.
container_window (int) – How many container items (such as a list in a list array) to preview at the begin and end of the array when the array is bigger than the window.
skip_new_lines (bool) – If the array should be rendered as a single line of text or if each element should be on its own line.
- tolist(self)¶
Alias of to_pylist for compatibility with NumPy.
- unique(self)¶
Compute distinct elements in array.
- Returns:
unique – An array of the same data type, with deduplicated elements.
- Return type:
- validate(self, *, full=False)¶
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).
- Parameters:
full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.
- Raises:
ArrowInvalid –
- value_counts(self)¶
Compute counts of unique elements in array.
- Returns:
An array of <input type “Values”, int64 “Counts”> structs
- Return type:
StructArray