砖与Conda 6.0运行时(不支持)
请注意
这个版本是不再可用。如果你想使用Conda来管理Python库和环境,使用支持版本机器学习的砖运行时。
砖运行时6.0 Conda(β)允许您利用Conda管理Python库和环境。这个运行时提供了两根Conda环境选择在创建集群:
砖的标准环境包括许多流行的Python包的更新版本。这个环境是作为现有笔记本电脑上运行的替代数据砖运行时。这是默认砖Conda-based运行时环境。
砖最小环境包含所需的最低数量的包PySpark和砖Python笔记本的功能。这个环境是理想的,如果你想自定义运行时各种Python包。
包括支持数据砖库工具(dbutils.library)。
请注意
Scala、Java和R在砖6.0运行时库Conda砖运行时的6.0是相同的。详情,请参阅砖运行时的6.0(不支持)发行说明。信息如何使用砖和Conda运行时,看到的砖与Conda运行时。
新功能
看到砖6.0运行时新功能。
改进
看到砖6.0运行时改进。
已知的问题
默认情况下,每个Python笔记本运行在自己的孤立Conda环境。这个孤立的环境是克隆从根Conda环境。因为克隆是一项昂贵的操作,在某些情况下,您可能会遇到以下问题:
如果集群实例类型没有本地存储,创建集群可能会失败的错误:
无法开始的火花。这可能发生在安装兼容库或初始化脚本时失败了。databricks_error_message:火花未能开始:超时后……秒
同时附加许多Python笔记本一个集群(例如,由安排工作或笔记本工作流)可能会导致一些笔记本无法连接。
以上如果您遇到任何问题,您不需要在隔离环境中运行Python笔记本(也就是说,您的集群不是共享),您可以禁用创建一个孤立的Python环境每个Python笔记本通过设置
spark.databricks.libraryIsolation.enabled来假在火花配置。设置这个标志也禁用dbutils.library。如果你升级安装Conda, Conda的新版本可能不包括修复Conda发行9104 (Conda列表失败如果“记录”文件有重复的条目)。如果你升级Conda看看失败将Python笔记本或使用
conda列表的错误TypeError:“<”不支持之间的实例的“NoneType”和str的在司机日志或笔记本,要么使用一个版本的Conda解决或避免升级Conda安装在这个版本。
库
以下是出口environment.yml文件默认根与Conda砖6.0运行时的环境。
砖的标准
的名字:databricks-standard渠道:- - - - - -违约依赖关系:- - - - - -_libgcc_mutex = 0.1 =主要- - - - - -asn1crypto = 0.24.0 = py37_0- - - - - -backcall = 0.1.0 = py37_0- - - - - -布拉斯特区= 1.0 = openblas- - - - - -宝途= 2.49.0 = py37_0- - - - - -boto3 = 1.9.162 = py_0- - - - - -botocore = 1.12.163 = py_0- - - - - -ca证书= 2019.1.23 = 0- - - - - -certifi = 2019.3.9 = py37_0- - - - - -cffi = 1.12.2 = py37h2e261b9_1- - - - - -chardet = 3.0.4 = py37_1003- - - - - -密码= 2.6.1 = py37h1ba5d50_0- - - - - -cython = 0.29.6 = py37he6710b0_0- - - - - -decorator = 4.4.0 = py37_1- - - - - -docutils = 0.14 = py37_0- - - - - -idna = 2.8 = py37_0- - - - - -ipython = 7.4.0 = py37h39e3cac_0- - - - - -ipython_genutils = 0.2.0 = py37_0- - - - - -绝地= 0.13.3 = py37_0- - - - - -jmespath = 0.9.4 = py_0- - - - - -krb5 = 1.16.1 = h173b8e3_7- - - - - -libedit = 3.1.20181209 = hc058e9b_0- - - - - -libffi = 3.2.1 = hd88cf55_4- - - - - -libgcc-ng = 8.2.0 = hdf63c60_1- - - - - -libgfortran-ng = 7.3.0 = hdf63c60_0- - - - - -libopenblas = 0.3.6 = h5a2b251_1- - - - - -libpq = 11.2 = h20c2e04_0- - - - - -libstdcxx-ng = 8.2.0 = hdf63c60_1- - - - - -ncurses = 6.1 = he6710b0_1- - - - - -nomkl = 3.0 = 0- - - - - -numpy = 1.16.2 = py37h99e49ec_0- - - - - -numpy-base = 1.16.2 = py37h2f8d375_0- - - - - -openssl = 1.1.1b = h7b6447c_1- - - - - -熊猫= 0.24.2 = py37he6710b0_0- - - - - -parso = 0.3.4 = py37_0- - - - - -容易受骗的人= 0.5.1 = py37_0- - - - - -pexpect = 4.6.0 = py37_0- - - - - -pickleshare = 0.7.5 = py37_0- - - - - -皮普= 19.0.3 = py37_0- - - - - -prompt_toolkit = 2.0.9 = py37_0- - - - - -psycopg2 = 2.7.6.1 = py37h1ba5d50_0- - - - - -ptyprocess = 0.6.0 = py37_0- - - - - -pycparser = 2.19 = py37_0- - - - - -pygments = 2.3.1 = py37_0- - - - - -pyopenssl = 19.0.0 = py37_0- - - - - -pysocks = 1.6.8 = py37_0- - - - - -python = 3.7.3 = h0371630_0- - - - - -python-dateutil = 2.8.0 = py37_0- - - - - -pytz = 2018.9 = py37_0- - - - - -readline = 7.0 = h7b6447c_5- - - - - -= 2.21.0 = py37_0请求- - - - - -s3transfer = 0.2.1 = py37_0- - - - - -scikit-learn = 0.20.3 = py37h22eb022_0- - - - - -scipy = 1.2.1 = py37he2b7bc3_0- - - - - -setuptools = 40.8.0 = py37_0- - - - - -6 = 1.12.0 = py37_0- - - - - -sqlite = 3.27.2 = h7b6447c_0- - - - - -statsmodels = 0.9.0 = py37h035aef0_0- - - - - -tk = 8.6.8 = hbc83047_0- - - - - -traitlets = 4.3.2 = py37_0- - - - - -urllib3 = 1.24.1 = py37_0- - - - - -wcwidth = 0.1.7 = py37_0- - - - - -轮= 0.33.1 = py37_0- - - - - -xz = 5.2.4 = h14c3975_4- - - - - -zlib = 1.2.11 = h7b6447c_3- - - - - -皮普:- - - - - -周期计= = 0.10.0- - - - - -kiwisolver = = 1.1.0- - - - - -matplotlib = = 3.0.3- - - - - -pyarrow = = 0.13.0- - - - - -pyparsing = = 2.4.2- - - - - -seaborn = = 0.9.0前缀:/砖/ conda / env / databricks-standard
砖最小
的名字:databricks-minimal渠道:- - - - - -违约依赖关系:- - - - - -_libgcc_mutex = 0.1 =主要- - - - - -backcall = 0.1.0 = py37_0- - - - - -布拉斯特区= 1.0 = openblas- - - - - -ca证书= 2019.1.23 = 0- - - - - -certifi = 2019.3.9 = py37_0- - - - - -decorator = 4.4.0 = py37_1- - - - - -ipython = 7.4.0 = py37h39e3cac_0- - - - - -ipython_genutils = 0.2.0 = py37_0- - - - - -绝地= 0.13.3 = py37_0- - - - - -libedit = 3.1.20181209 = hc058e9b_0- - - - - -libffi = 3.2.1 = hd88cf55_4- - - - - -libgcc-ng = 8.2.0 = hdf63c60_1- - - - - -libgfortran-ng = 7.3.0 = hdf63c60_0- - - - - -libopenblas = 0.3.6 = h5a2b251_1- - - - - -libstdcxx-ng = 8.2.0 = hdf63c60_1- - - - - -ncurses = 6.1 = he6710b0_1- - - - - -nomkl = 3.0 = 0- - - - - -numpy = 1.16.2 = py37h99e49ec_0- - - - - -numpy-base = 1.16.2 = py37h2f8d375_0- - - - - -openssl = 1.1.1b = h7b6447c_1- - - - - -熊猫= 0.24.2 = py37he6710b0_0- - - - - -parso = 0.3.4 = py37_0- - - - - -pexpect = 4.6.0 = py37_0- - - - - -pickleshare = 0.7.5 = py37_0- - - - - -皮普= 19.0.3 = py37_0- - - - - -prompt_toolkit = 2.0.9 = py37_0- - - - - -ptyprocess = 0.6.0 = py37_0- - - - - -pygments = 2.3.1 = py37_0- - - - - -python = 3.7.3 = h0371630_0- - - - - -python-dateutil = 2.8.0 = py37_0- - - - - -pytz = 2018.9 = py37_0- - - - - -readline = 7.0 = h7b6447c_5- - - - - -setuptools = 40.8.0 = py37_0- - - - - -6 = 1.12.0 = py37_0- - - - - -sqlite = 3.27.2 = h7b6447c_0- - - - - -tk = 8.6.8 = hbc83047_0- - - - - -traitlets = 4.3.2 = py37_0- - - - - -wcwidth = 0.1.7 = py37_0- - - - - -轮= 0.33.1 = py37_0- - - - - -xz = 5.2.4 = h14c3975_4- - - - - -zlib = 1.2.11 = h7b6447c_3- - - - - -皮普:- - - - - -pyarrow = = 0.13.0前缀:/砖/ conda / env / databricks-minimal