dtale 1.7.0

Python Terminal

This comes courtesy of PyCharm 👁 image8
Feel free to invoke python or ipython directly and use the commands in the screenshot above and it should work

import dtaleimport pandas as pddf = pd.DataFrame([dict(a=1,b=2,c=3)])# Assigning a reference to a running D-Tale processd = dtale.show(df)# Accessing data associated with D-Tale processtmp = d.data.copy()tmp['d'] = 4# Altering data associated with D-Tale process# FYI: this will clear any front-end settings you have at the time for this process (filter, sorts, formatting)d.data = tmp# Shutting down D-Tale processd.kill()# using Python's `webbrowser` package it will try and open your server's default browser to this processd.open_browser()# There is also some helpful metadata about the processd._data_id # the process's data identifierd._url # the url to access the processd2 = dtale.get_instance(d._data_id) # returns a new reference to the instance running at that data_iddtale.instances() # returns a dictionary of all instances available, this would be { 1: ... }

Jupyter Notebook

Within any jupyter (ipython) notebook executing a cell like this will display a small instance of D-Tale in the output cell. Here are some examples:

If you are running ipython<=5.0 then you also have the ability to adjust the size of your output cell for the most recent instance displayed:

👁 image12

One thing of note is that alot of the modal popups you see in the standard browser version will now open separate browser windows for spacial convienence:

Command-line

Base CLI options (run dtale --help to see all options available)

Loading data from arctic

dtale--arctic-hostmongodb://localhost:27027--arctic-libraryjdoe.my_lib--arctic-nodemy_node--arctic-start20130101--arctic-end20161231

Loading data from CSV

dtale--csv-path/home/jdoe/my_csv.csv--csv-parse_datesdate

Loading data from a Custom loader - Using the DTALE_CLI_LOADERS environment variable, specify a path to a location containing some python modules - Any python module containing the global variables LOADER_KEY & LOADER_PROPS will be picked up as a custom loader - LOADER_KEY: the key that will be associated with your loader. By default you are given arctic & csv (if you use one of these are your key it will override these) - LOADER_PROPS: the individual props available to be specified. - For example, with arctic we have host, library, node, start & end. - If you leave this property as an empty list your loader will be treated as a flag. For example, instead of using all the arctic properties we would simply specify --arctic (this wouldn’t work well in arctic’s case since it depends on all those properties) - You will also need to specify a function with the following signature def find_loader(kwargs) which returns a function that returns a dataframe or None - Here is an example of a custom loader:

from dtale.cli.clickutils import get_loader_options'''
 IMPORTANT!!! This global variable is required for building any customized CLI loader.
 When find loaders on startup it will search for any modules containing the global variable LOADER_KEY.
'''LOADER_KEY = 'testdata'LOADER_PROPS = ['rows', 'columns']def test_data(rows, columns): import pandas as pd import numpy as np import random from past.utils import old_div from pandas.tseries.offsets import Day from dtale.utils import dict_merge import string now = pd.Timestamp(pd.Timestamp('now').date()) dates = pd.date_range(now - Day(364), now) num_of_securities = max(old_div(rows, len(dates)), 1) # always have at least one security securities = [ dict(security_id=100000 + sec_id, int_val=random.randint(1, 100000000000), str_val=random.choice(string.ascii_letters) * 5) for sec_id in range(num_of_securities) ] data = pd.concat([ pd.DataFrame([dict_merge(dict(date=date), sd) for sd in securities]) for date in dates ], ignore_index=True)[['date', 'security_id', 'int_val', 'str_val']] col_names = ['Col{}'.format(c) for c in range(columns)] return pd.concat([data, pd.DataFrame(np.random.randn(len(data), columns), columns=col_names)], axis=1)# IMPORTANT!!! This function is required for building any customized CLI loader.def find_loader(kwargs): test_data_opts = get_loader_options(LOADER_KEY, kwargs) if len([f for f in test_data_opts.values() if f]): def _testdata_loader(): return test_data(int(test_data_opts.get('rows', 1000500)), int(test_data_opts.get('columns', 96))) return _testdata_loader return None

In this example we simplying building a dataframe with some dummy data based on dimensions specified on the command-line: - --testdata-rows - --testdata-columns

Here’s how you would use this loader:

DTALE_CLI_LOADERS=./path_to_loadersbash-c'dtale --testdata-rows 10 --testdata-columns 5'

Dimensions/Main Menu

The information in the upper right-hand corner gives grid dimensions 👁 image20
- lower-left => row count - upper-right => column count - clicking the triangle displays the menu of standard functions (click outside menu to close it) 👁 image21

Selecting/Deselecting Columns

• to select a column, simply click on the column header (to deselect, click the column header again)

  • You’ll notice that the columns you’ve selected will display in the top of your browser 👁 image22
    

Menu functions w/ no columns selected

👁 image23

import pandas as pdimport dtaledtale.show(pd.DataFrame([dict(foo=1, bar=2, biz=3, baz=4, snoopy_D_O_double_gizzle=5)]))dtale.show(pd.DataFrame([ dict(a=1, b=2, c=3, d=4), dict(a=2, b=3, c=4, d=5), dict(a=3, b=4, c=5, d=6), dict(a=4, b=5, c=6, d=7)]))dtale.show(pd.DataFrame([range(6), range(6), range(6), range(6), range(6), range(6)]), name="foo")

Menu functions w/ column(s) selected

👁 image59

Menu functions within a Jupyter Notebook

These are the same functions as the menu listed earlier, but there is no more column selection (instead theres menus for each column). Also the following buttons will no longer open modals, but separate browser windows: Correlations, Describe & Instances (see images from Jupyter Notebook, also Charts will always open in a separate browser window)

There are also menus associated with each column header which can be trigger by clicking on a column header. The functions that are contained within each are: Sorting, Move To Front, Lock/Unlock, Histogram, Describe, Formats (see image from Jupyter Notebook) - Histogram & Describe open separate browser windows

Cloning

Clone the code (git clone ), then start the backend server:

$gitclonessh://git@github.com:manahl/dtale.git# install the dependencies
$pythonsetup.pydevelop# start the server
$pythondtale--csv-path/home/jdoe/my_csv.csv--csv-parse_datesdate

You can also run dtale from PyDev directly.

You will also want to import javascript dependencies and build the source:

$npminstall# 1) a persistent server that serves the latest JS:
$npmrunwatch# 2) or one-off build:
$npmrunbuild

Running tests

The usual npm test command works:

$ npm test

You can run individual test files:

$ TEST=static/__tests__/dtale/DataViewer-base-test.jsx npm run test-file

Linting

You can lint all the JS and CSS to confirm there’s nothing obviously wrong with it:

$npmrunlint-s

You can also lint individual JS files:

$npmrunlint-js-file-s--static/dtale/DataViewer.jsx

Formatting JS

You can auto-format code as follows:

$npmrunformat

Docker Development

You can build python 27-3 & run D-Tale as follows:

$yarnrunbuild$docker-composebuilddtale_2_7$dockerrun-it--networkhostdtale_2_7:latest$python>>>importpandasaspd>>>df=pd.DataFrame([dict(a=1,b=2,c=3)])>>>importdtale>>>dtale.show(df)

Then view your D-Tale instance in your browser using the link that gets printed

You can build python 36-1 & run D-Tale as follows:

$yarnrunbuild$docker-composebuilddtale_3_6$dockerrun-it--networkhostdtale_3_6:latest$python>>>importpandasaspd>>>df=pd.DataFrame([dict(a=1,b=2,c=3)])>>>importdtale>>>dtale.show(df)

Then view your D-Tale instance in your browser using the link that gets printed

1.7.0 (2020-1-28)

• redesign of charts popup to use plotly/dash

• #55: raise exception when data contains duplicate column names

• heatmap integration

• combination of “_main.jsx” files into one for spacial optimization

• #15: made arctic an “extra” dependency

1.6.10 (2020-1-12)

• better front-end handling of dates for charting as to avoid timezone issues

• the ability to switch between sorting any axis in bar charts

1.6.9 (2020-1-9)

• bugfix for timezone issue around passing date filters to server for scatter charts in correlations popup

1.6.8 (2020-1-9)

• additional information about how to use Correlations popup

• handling of all-nan data in charts popup

• styling issues on popups (especially Histogram)

• removed auto-filtering on correlation popup

• scatter point color change

• added chart icon to cell that has been selected in correlation popup

• responsiveness to scatter charts

• handling of links to ‘main’,‘iframe’ & ‘popup’ missing data_id

• handling of ‘inf’ values when getting min/max & describe data

• added header to window popups (correlations, charts, …) and a link back to the grid

• added egg building to cirleci script

• correlation timeseries chart hover line

1.6.7 (2020-1-3)

• #50: updates to rolling correlation functionality

1.6.6 (2020-1-2)

• #47: selection of multiple columns for y-axis

• updated histogram bin selection to be an input box for full customization

• better display of timestamps in axis ticks for charts

• sorting of bar charts by y-axis

• #48: scatter charts in chart builder

• “nunique” added to list of aggregations

• turned on “threaded=True” for app.run to avoid hanging popups

• #45: rolling computations as aggregations

• Y-Axis editor

1.6.5 (2019-12-29)

• test whether filters entered will return no data and block the user from apply those

• allow for group values of type int or float to be displayed in charts popup

• timeseries correlation values which return ‘nan’ will be replaced by zero for chart purposes

• update ‘distribution’ to ‘series’ on charts so that missing dates will not show up as ticks

• added “fork on github” flag for demo version & links to github/docs on “About” popup

• limited lz4 to <= 2.2.1 in python 27-3 since latest version is no longer supported

1.6.4 (2019-12-26)

• testing of hostname returned by socket.gethostname, use ‘localhost’ if it fails

• removal of flask dev server banner when running in production environments

• better handling of long strings in wordclouds

• #43: only show timeseries correlations if datetime columns exist with multiple values per date

1.6.3 (2019-12-23)

• updated versions of packages in yarn.lock due to issue with chart.js box & whisker plots

1.6.2 (2019-12-23)

• #40: loading initial chart as non-line in chart builder

• #41: double clicking cells in correlation grid for scatter will cause chart not to display

• “Open Popup” button for ipython iframes

• column width resizing on sorting

• additional int/float descriptors (sum, median, mode, var, sem, skew, kurt)

• wordcloud chart type

1.6.1 (2019-12-19)

• bugfix for url display when running from command-line

1.6.0 (2019-12-19)

• charts integration

  • the ability to look at data in line, bar, stacked bar & pie charts

  • the ability to group & aggregate data within the charts

• direct ipython iframes to correlations & charts pages with pre-selected inputs

• the ability to access instances from code by data id dtale.get_instance(data_id)

• view all active data instances dtale.instances()

1.5.1 (2019-12-12)

• conversion of new flask instance for each dtale.show call to serving all data associated with one parent process under the same flask instance unless otherwise specified by the user (the force parameter)

1.5.0 (2019-12-02)

• ipython integration

  • ipython output cell adjustment

  • column-wise menu support

  • browser window popups for: Correlations, Coverage, Describe, Histogram & Instances

1.4.1 (2019-11-20)

• #32: unpin jsonschema by moving flasgger to extras_require

1.4.0 (2019-11-19)

• Correlations Pearson Matrix filters

• “name” display in title tab

• “Heat Map” toggle

• dropped unused “Flask-Caching” requirement

1.3.7 (2019-11-12)

• Bug fixes for:

  • #28: “Instances” menu option will now be displayed by default

  • #29: add hints to how users can navigate the correlations popup

  • add “unicode” as a string classification for column width calculation

1.3.6 (2019-11-08)

• Bug fixes for:

  • choose between pandas.corr & numpy.corrcoef depending on presence of NaNs

  • hide timeseries correlations when date columns only contain one day

1.3.5 (2019-11-07)

• Bug fixes for:

  • duplicate loading of histogram data

  • string serialization failing when mixing future.str & str in scatter function

1.3.4 (2019-11-07)

• updated correlation calculation to use numpy.corrcoef for performance purposes

• github rebranding from manahl -> man-group

1.3.3 (2019-11-05)

• hotfix for failing test under certain versions of future package

1.3.2 (2019-11-05)

• Bug fixes for:

  • display of histogram column information

  • reload of hidden “processes” input when loading instances data

  • correlations json failures on string conversion

1.3.1 (2019-10-29)

• fix for incompatible str types when directly altering state of data in running D-Tale instance

1.3.0 (2019-10-29)

• webbrowser integration (the ability to automatically open a webbrowser upon calling dtale.show())

• flag for hiding the “Shutdown” button for long-running demos

• “Instances” navigator popup for viewing all activate D-Tale instances for the current python process

1.2.0 (2019-10-24)

• #20: fix for data being overriden with each new instance

• #21: fix for displaying timestamps if they exist

• calling show() now returns an object which can alter the state of a process

  • accessing/altering state through the data property

  • shutting down a process using the kill() function

1.1.1 (2019-10-23)

• #13: fix for auto-detection of column widths for strings and floats

1.1.0 (2019-10-08)

• IE support

• Describe & About popups

• Custom CLI support

1.0.0 (2019-09-06)

• Initial public release