Databricks Runtime 7.2 for ML(不支持)
Databricks于2020年8月发布了这张图片。
Databricks Runtime 7.2 for Machine Learning为机器学习和数据科学提供了一个现成的环境Databricks Runtime 7.2(不支持).Databricks Runtime ML包含许多流行的机器学习库,包括TensorFlow、PyTorch和XGBoost。它还支持使用Horovod进行分布式深度学习训练。
有关更多信息,包括创建Databricks Runtime ML集群的说明,请参见Databricks运行时机器学习.
新特性和主要变化
Databricks Runtime 7.2 ML是建立在Databricks Runtime 7.2之上的。有关Databricks Runtime 7.2(包括Apache Spark MLlib和SparkR)中的新特性的信息,请参见Databricks Runtime 7.2(不支持)发行说明。
Databricks Runtime ML Python环境的主要更改
本节描述与已安装的Databricks Runtime ML Python环境相比的主要变化Databricks Runtime 7.1 for ML(不支持).您还应该查看Databricks Runtime Python环境的主要更改Databricks Runtime 7.2(不支持).有关已安装的Python包及其版本的完整列表,请参见Python库.
改进
Databricks Runtime 7.2改进了对TensorBoard的支持。看到Databricks Runtime 7.2发布说明.
系统环境
Databricks Runtime 7.2 ML的系统环境与Databricks Runtime 7.2不同:
DBUtils: Databricks Runtime ML不包含库实用程序(dbutls . Library).你可以使用
%皮普
而且% conda
而不是命令。看到笔记本范围的Python库.对于GPU集群,Databricks Runtime ML包含以下NVIDIA GPU库:
CUDA 10.1更新2
cuDNN 7.6.5
NCCL 2.7.3
TensorRT 6.0.1中
库
以下部分列出了Databricks Runtime 7.2 ML中包含的不同于Databricks Runtime 7.2的库。
Python库
Databricks Runtime 7.2 ML使用Conda进行Python包管理,包括许多流行的ML包。下面介绍Databricks Runtime 7.2 ML的Conda环境。
CPU集群上的Python
的名字:databricks-ml渠道:-pytorch-违约依赖关系:-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_1 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-补丁= 1.0 = py_2-bcrypt = 3.1.7 = py37h7b6447c_1-布拉斯特区= 1.0 = mkl-信号灯= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.15.0 = h7b6447c_1001-ca证书= 2020.6.24 = 0-cachetools = 4.1.0 = py_1-certifi = 2020.6.20 = py37_0-cffi = 1.14.0 = py37h2e261b9_0-chardet = 3.0.4 = py37_1003-单击= 7.0 = py37_0-= py_0 1.3.0 cloudpickle =版本-configparser = 3.7.4 = py37_0-cpuonly = 1.0 = 0-密码= 2.8 = py37h1ba5d50_0-周期计= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1装饰= = py_0-莳萝= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未来= 0.18.2 = py37_1-恐吓= 0.3.3 = py_0-gitdb =你= py_0-gitpython = 3.1.0 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = py37_0-isodate = 0.6.0 = py_1-itsdangerous = 1.1.0 = py37_0-绝地= 0.14.1 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.10.0 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.16.4 = h173b8e3_0-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.2.1 = hd88cf55_4-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 11.2 = h20c2e04_0-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鲭鲨= 1.1.2 = py_0-减价= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h14c3975_1-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_1-忍者= 1.9.0 = py37hfd86e86_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1g = h7b6447c_0-包装= 20.1 = py_0-熊猫= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.5.2 = py_0-容易受骗的人= 0.5.1 = py37_0-pexpect = 4.8.0 = py37_1-pickleshare = 0.7.5 = py37_1001-枕头= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-情节= 4.8.2 = py_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.4 = py37h1ba5d50_0-ptyprocess = 0.6.0 = py37_0-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.7 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = py_1-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_1-python =第3.7.6 = h0371630_2-python-dateutil = 2.8.1发布= py_0-python编辑器的1.0.4 = = py_0-pytorch = 1.5.1 = py3.7_cpu_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 7.0 = h7b6447c_5-= 2.22.0 = py37_1请求-= py_0 1.3.0 requests-oauthlib =版本-重试= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_1-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37_0-smmap = 3.0.2 = py_0-sqlite = 3.31.1 = h62c20be_1-sqlparse = 0.3.0 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-汇总= 0.8.3 = py37_0-tk = 8.6.8 = hbc83047_0-torchvision = 0.6.1 = py37_cpu-龙卷风= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-轮= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.4 = h14c3975_4-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普:-astunparse = = 1.6.3-azure-core = = 1.7.0-azure-storage-blob = = 12.3.2-databricks-cli = = 0.11.0-diskcache = = 4.1.0-码头工人= = 4.2.2-大猩猩= = 0.3.0-horovod = = 0.19.5-hyperopt = = 0.2.4.db2-joblibspark = = 0.2.0-keras-preprocessing = = 1.1.2-考拉= = 1.1.0-mleap = = 0.16.1-mlflow = = 1.9.1-msr = = 0.6.17-opt-einsum = = 3.3.0-petastorm = = 0.9.2-pyarrow = = 0.15.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-spark-tensorflow-distributor = = 0.1.0-sparkdl = = 2.1.0-db1-2.2.2 tensorboard = =-tensorboard-plugin-wit = = 1.7.0-tensorflow-cpu = = 2.2.0-tensorflow-estimator = = 2.2.0-termcolor = = 1.1.0-xgboost = = 1.1.1前缀:/砖/ conda / env / databricks-ml
GPU集群下的Python
的名字:databricks-ml-gpu渠道:-pytorch-违约依赖关系:-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_1 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-补丁= 1.0 = py_2-bcrypt = 3.1.7 = py37h7b6447c_1-布拉斯特区= 1.0 = mkl-信号灯= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.15.0 = h7b6447c_1001-ca证书= 2020.6.24 = 0-cachetools = 4.1.0 = py_1-certifi = 2020.6.20 = py37_0-cffi = 1.14.0 = py37h2e261b9_0-chardet = 3.0.4 = py37_1003-单击= 7.0 = py37_0-= py_0 1.3.0 cloudpickle =版本-configparser = 3.7.4 = py37_0-密码= 2.8 = py37h1ba5d50_0-cudatoolkit = 10.1.243 = h6bb024c_0-周期计= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1装饰= = py_0-莳萝= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未来= 0.18.2 = py37_1-恐吓= 0.3.3 = py_0-gitdb =你= py_0-gitpython = 3.1.0 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = py37_0-isodate = 0.6.0 = py_1-itsdangerous = 1.1.0 = py37_0-绝地= 0.14.1 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.10.0 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.16.4 = h173b8e3_0-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.2.1 = hd88cf55_4-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 11.2 = h20c2e04_0-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鲭鲨= 1.1.2 = py_0-减价= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h14c3975_1-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_1-忍者= 1.9.0 = py37hfd86e86_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1g = h7b6447c_0-包装= 20.1 = py_0-熊猫= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.5.2 = py_0-容易受骗的人= 0.5.1 = py37_0-pexpect = 4.8.0 = py37_1-pickleshare = 0.7.5 = py37_1001-枕头= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-情节= 4.8.2 = py_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.4 = py37h1ba5d50_0-ptyprocess = 0.6.0 = py37_0-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.7 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = py_1-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_1-python =第3.7.6 = h0371630_2-python-dateutil = 2.8.1发布= py_0-python编辑器的1.0.4 = = py_0-pytorch = 1.5.1 = py3.7_cuda10.1.243_cudnn7.6.3_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 7.0 = h7b6447c_5-= 2.22.0 = py37_1请求-= py_0 1.3.0 requests-oauthlib =版本-重试= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_1-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37_0-smmap = 3.0.2 = py_0-sqlite = 3.31.1 = h62c20be_1-sqlparse = 0.3.0 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-汇总= 0.8.3 = py37_0-tk = 8.6.8 = hbc83047_0-torchvision = 0.6.1 = py37_cu101-龙卷风= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-轮= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.4 = h14c3975_4-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普:-astunparse = = 1.6.3-azure-core = = 1.7.0-azure-storage-blob = = 12.3.2-databricks-cli = = 0.11.0-diskcache = = 4.1.0-码头工人= = 4.2.2-大猩猩= = 0.3.0-horovod = = 0.19.5-hyperopt = = 0.2.4.db2-joblibspark = = 0.2.0-keras-preprocessing = = 1.1.2-考拉= = 1.1.0-mleap = = 0.16.1-mlflow = = 1.9.1-msr = = 0.6.17-opt-einsum = = 3.3.0-petastorm = = 0.9.2-pyarrow = = 0.15.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-spark-tensorflow-distributor = = 0.1.0-sparkdl = = 2.1.0-db1-2.2.2 tensorboard = =-tensorboard-plugin-wit = = 1.7.0-tensorflow = = 2.2.0-tensorflow-estimator = = 2.2.0-termcolor = = 1.1.0-xgboost = = 1.1.1前缀:/砖/ conda / env / databricks-ml-gpu
Java和Scala库(Scala 2.12集群)
除了Java和Scala库在Databricks Runtime 7.2, Databricks Runtime 7.2 ML包含以下jar:
组ID |
工件ID |
版本 |
---|---|---|
com.typesafe.akka |
akka-actor_2.12 |
2.5.23 |
ml.combust.mleap |
mleap-databricks-runtime_2.12 |
0.17.2-4882dc3 |
ml.dmlc |
xgboost4j-spark_2.12 |
1.0.0 |
ml.dmlc |
xgboost4j_2.12 |
1.0.0 |
org.mlflow |
mlflow-client |
1.9.1 |
org.scala-lang.modules |
scala-java8-compat_2.12 |
0.8.0 |
org.tensorflow |
spark-tensorflow-connector_2.12 |
1.15.0 |