Databricks Runtime 7.0 ML(不支持)

Databricks于2020年6月发布了这张图片。

Databricks Runtime 7.0 for Machine Learning为机器学习和数据科学提供了一个现成的环境Databricks Runtime 7.0(不支持)．Databricks Runtime ML包含许多流行的机器学习库，包括TensorFlow、PyTorch和XGBoost。它还支持使用Horovod进行分布式深度学习训练。

有关更多信息，包括创建Databricks Runtime ML集群的说明，请参见Databricks运行时机器学习．

新特性和主要变化

Databricks Runtime 7.0 ML是建立在Databricks Runtime 7.0之上的。有关Databricks Runtime 7.0(包括Apache Spark MLlib和SparkR)中的新特性的信息，请参见Databricks Runtime 7.0(不支持)发行说明。

GPU-aware调度

Databricks Runtime 7.0 ML支持来自Apache Spark 3.0的gpu感知调度。Databricks自动为您配置。看到GPU调度．

ML Python环境的主要变化

本节描述与预安装的ML Python环境相比的主要变化Databricks Runtime 6.6 ML(不支持)．中基本Python环境的主要更改Databricks Runtime 7.0(不支持)．有关已安装的Python包及其版本的完整列表，请参见Python库．

Python包升级

  • Tensorflow 1.15.0 -> 2.2.0

  • Tensorboard 1.15.0 -> 2.2.2

  • Pytorch 1.4.0 -> 1.5.0

  • Xgboost 0.90 -> 1.1.1

  • Sparkdl 1.6.0-db1 -> 2.1.0-db1

  • hyperopt 0.2.2。Db1 -> 0.2.4.1 Db1

新增Python包

  • lightgbm: tripwire

  • nltk: 3.4.5

  • petastorm: 0.9.2

  • 4.5.2情节:

删除Python包

  • argparse

  • 宝途(使用boto3相反)

  • 彩色光

  • 弃用

  • et-xmlfile

  • fusepy

  • html5lib

  • jdcal

  • keras(使用tensorflow.keras相反)

  • keras-applications(使用tensorflow.keras.applications相反)

  • llvmlite

  • lxml

  • 鼻子

  • nose-exclude

  • numba

  • openpyxl

  • pathlib2

  • 厚度

  • pymongo

  • singledispatch

  • tensorboardX(使用torch.utils.tensorboard相反)

  • virtualenv

  • webencodings

ML R环境的重大变化

Databricks Runtime 7.0 ML包含RStudio Server Open Source v1.2.5033的未修改版本，源代码可以bob下载地址在GitHub．阅读更多RStudio服务器在砖上。

ML Spark包、Java和Scala库的更改

升级如下包。有些升级为快照与Apache Spark 3.0兼容的版本:

  • 图帧:0.7.0- db2- spark2.4 -> 0.8.0-db2-spark3.0

  • spark-tensorflow-connector: 1.15.0 (Scala 2.11) -> (Scala 2.12)

  • Xgboost4j和Xgboost4j -spark: 0.90 -> 1.0.0

  • mleap- databicks -runtime: 0.17.0-4882dc3(快照)

删除以下包:

  • TensorFlow (Java)

  • TensorFrames

  • Apache Spark的深度学习管道(HorovodRunner在Python中可用)

增加了conda和pip命令来支持笔记本范围的Python库(公开预览)

从Databricks Runtime 7.0 ML开始，您可以使用%皮普而且% conda命令来管理安装在notebook会话中的Python库。您还可以使用这些命令为笔记本电脑创建自定义环境，并在笔记本电脑之间重现此环境。若要启用此功能，请在集群设置中设置火花配置spark.databricks.conda.condaMagic.enabled真正的．有关更多信息，请参见笔记本范围的Python库．

弃用和不支持的特性

Databricks Runtime 7.0 ML不支持表访问控制．如果您需要表访问控制，我们建议您使用Databricks Runtime 7.0。

已知的问题

  • 通过sample_input参数mlflow.spark.log_model为了记录mleap格式的MLlib模型，由于mleap API的更改而失败，导致AttributeError。升级到MLflow 1.9.0作为解决方案。可以使用以下命令安装MLflow 1.9.0笔记本范围的Python库或工作区库

系统环境

Databricks Runtime 7.0 ML的系统环境与Databricks Runtime 7.0不同:

库

以下部分列出了Databricks Runtime 7.0 ML中包含的不同于Databricks Runtime 7.0的库。

Python库

Databricks Runtime 7.0 ML使用Conda进行Python包管理，包括许多流行的ML包。下面介绍Databricks Runtime 7.0 ML的Conda环境。

CPU集群上的Python

的名字：databricks-ml渠道：-pytorch-违约依赖关系：-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_0 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-补丁= 1.0 = py_2-bcrypt = 3.1.7 = py37h7b6447c_1-布拉斯特区= 1.0 = mkl-信号灯= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.15.0 = h7b6447c_1001-ca证书= 2020.1.1 = 0-cachetools = 4.1.0 = py_1-certifi = 2020.4.5.1 = py37_0-cffi = 1.14.0 = py37h2e261b9_0-chardet = 3.0.4 = py37_1003-单击= 7.0 = py37_0-= py_0 1.3.0 cloudpickle =版本-configparser = 3.7.4 = py37_0-cpuonly = 1.0 = 0-密码= 2.8 = py37h1ba5d50_0-周期计= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1装饰= = py_0-莳萝= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未来= 0.18.2 = py37_1-恐吓= 0.3.3 = py_0-gitdb2 = 2.0.6 = py_0-gitpython = 3.0.5 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = py37_0-itsdangerous = 1.1.0 = py37_0-绝地= 0.14.1 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.9.4 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.16.4 = h173b8e3_0-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.2.1 = hd88cf55_4-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 11.2 = h20c2e04_0-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鲭鲨= 1.1.2 = py_0-减价= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h7b6447c_0-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_0-忍者= 1.9.0 = py37hfd86e86_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1g = h7b6447c_0-包装= 20.1 = py_0-熊猫= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.5.2 = py_0-容易受骗的人= 0.5.1 = py37_0-pexpect = 4.8.0 = py37_0-pickleshare = 0.7.5 = py37_0-枕头= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-4.5.2情节= = py_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.4 = py37h1ba5d50_0-ptyprocess = 0.6.0 = py37_0-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.7 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = py37_0-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_0-python =第3.7.6 = h0371630_2-python-dateutil = 2.8.1发布= py_0-python编辑器的1.0.4 = = py_0-pytorch = 1.5.0 = py3.7_cpu_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 7.0 = h7b6447c_5-= 2.22.0 = py37_1请求-= py_0 1.3.0 requests-oauthlib =版本-重试= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_0-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37_0-smmap2 = 2.0.5 = py37_0-sqlite = 3.31.1 = h62c20be_1-sqlparse = 0.3.0 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-汇总= 0.8.3 = py37_0-tk = 8.6.8 = hbc83047_0-torchvision = 0.6.0 = py37_cpu-龙卷风= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-轮= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.4 = h14c3975_4-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普：-astunparse = = 1.6.3-databricks-cli = = 0.11.0-diskcache = = 4.1.0-码头工人= = 4.2.1-大猩猩= = 0.3.0-horovod = = 0.19.1-hyperopt = = 0.2.4.db1-keras-preprocessing = = 1.1.2-mleap = = 0.16.0-mlflow = = 1.8.0-opt-einsum = = 3.2.1之上-petastorm = = 0.9.2-pyarrow = = 0.15.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-sparkdl = = 2.1.0-db1-2.2.2 tensorboard = =-tensorboard-plugin-wit = = 1.6.0.post3-tensorflow-cpu = = 2.2.0-tensorflow-estimator = = 2.2.0-termcolor = = 1.1.0-xgboost = = 1.1.1前缀：/砖/ conda / env / databricks-ml

GPU集群下的Python

的名字：databricks-ml-gpu渠道：-pytorch-违约依赖关系：-_libgcc_mutex = 0.1 =主要-absl-py = 0.9.0 = py37_0-= py37_0 1.3.0 asn1crypto =版本-阿斯特= 0.8.0 = py37_0-backcall = 0.1.0 = py37_0-补丁= 1.0 = py_2-bcrypt = 3.1.7 = py37h7b6447c_1-布拉斯特区= 1.0 = mkl-信号灯= 1.4 = py37_0-boto3 = 1.12.0 = py_0-botocore = 1.15.0 = py_0-c-ares = 1.15.0 = h7b6447c_1001-ca证书= 2020.1.1 = 0-cachetools = 4.1.0 = py_1-certifi = 2020.4.5.2 = py37_0-cffi = 1.14.0 = py37h2e261b9_0-chardet = 3.0.4 = py37_1003-单击= 7.0 = py37_0-= py_0 1.3.0 cloudpickle =版本-configparser = 3.7.4 = py37_0-密码= 2.8 = py37h1ba5d50_0-cudatoolkit = 10.1.243 = h6bb024c_0-周期计= 0.10.0 = py37_0-cython = 0.29.15 = py37he6710b0_0-4.4.1装饰= = py_0-莳萝= 0.3.1.1 = py37_1-docutils = 0.15.2 = py37_0-entrypoints = 0.3 = py37_0-瓶= 1.1.1 = py_1-freetype的= 2.9.1 = h8a8886c_1-未来= 0.18.2 = py37_1-恐吓= 0.3.3 = py_0-gitdb2 = 2.0.6 = py_0-gitpython = 3.0.5 = py_0-google-auth = 1.11.2 = py_0-google-auth-oauthlib = 0.4.1 = py_2-google-pasta = 0.2.0 = py_0-grpcio = 1.27.2 = py37hf8bcb03_0-gunicorn = 20.0.4 = py37_0-h5py = 2.10.0 = py37h7918eee_0-hdf5 = 1.10.4 = hb1b8bf9_0-icu = 58.2 = he6710b0_3-idna = 2.8 = py37_0-intel-openmp = 2020.0 = 166-ipykernel = 5.1.4 = py37h39e3cac_0-ipython = 7.12.0 = py37h5ca1d4c_0-ipython_genutils = 0.2.0 = py37_0-itsdangerous = 1.1.0 = py37_0-绝地= 0.14.1 = py37_0-jinja2 = 2.11.1 = py_0-jmespath = 0.9.4 = py_0-joblib = 0.14.1 = py_0-jpeg = 9 b = h024ee3a_2-jupyter_client = 5.3.4 = py37_0-jupyter_core = 4.6.1 = py37_0-kiwisolver = 1.1.0 = py37he6710b0_0-krb5 = 1.16.4 = h173b8e3_0-ld_impl_linux - 64 = 2.33.1 = h53a641e_7-libedit = 3.1.20181209 = hc058e9b_0-libffi = 3.2.1 = hd88cf55_4-libgcc-ng = 9.1.0 = hdf63c60_0-libgfortran-ng = 7.3.0 = hdf63c60_0-libpng = 1.6.37 = hbc83047_0-libpq = 11.2 = h20c2e04_0-libprotobuf = 3.11.4 = hd408876_0-libsodium = 1.0.16 = h1bed415_0-libstdcxx-ng = 9.1.0 = hdf63c60_0-libtiff = 4.1.0 = h2733197_0-lightgbm = tripwire = py37he6710b0_0-lz4-c = 1.8.1.2 = h14c3975_0-尖吻鲭鲨= 1.1.2 = py_0-减价= 3.1.1 = py37_0-markupsafe = 1.1.1 = py37h7b6447c_0-matplotlib-base = 3.1.3 = py37hef1b27d_0-mkl = 2020.0 = 166-mkl-service = tripwire = py37he904b0f_0-mkl_fft = 1.0.15 = py37ha843d7b_0-mkl_random = 1.1.0 = py37hd6b4f25_0-ncurses = 6.2 = he6710b0_1-networkx = 2.4 = py_0-忍者= 1.9.0 = py37hfd86e86_0-nltk = 3.4.5 = py37_0-numpy = 1.18.1 = py37h4f9e942_0-numpy-base = 1.18.1 = py37hde5b4d6_1-oauthlib = 3.1.0 = py_0-olefile = 0.46 = py37_0-openssl = 1.1.1g = h7b6447c_0-包装= 20.1 = py_0-熊猫= 1.0.1 = py37h0573a6f_0-paramiko = 2.7.1 = py_0-parso = 0.5.2 = py_0-容易受骗的人= 0.5.1 = py37_0-pexpect = 4.8.0 = py37_0-pickleshare = 0.7.5 = py37_0-枕头= 7.0.0 = py37hb39fc2d_0-皮普= 20.0.2 = py37_3-4.5.2情节= = py_0-prompt_toolkit = 3.0.3 = py_0-protobuf = 3.11.4 = py37he6710b0_0-psutil = 5.6.7 = py37h7b6447c_0-psycopg2 = 2.8.4 = py37h1ba5d50_0-ptyprocess = 0.6.0 = py37_0-pyasn1 = 0.4.8 = py_0-pyasn1-modules = 0.2.7 = py_0-pycparser = 2.19 = py37_0-pygments = 2.5.2 = py_0-pyjwt = 1.7.1上= py37_0-= py37h7b6447c_0 1.3.0 pynacl =版本-pyodbc = 4.0.30 = py37he6710b0_0-pyopenssl = 19.1.0 = py37_0-pyparsing = 2.4.6 = py_0-pysocks = 1.7.1上= py37_0-python =第3.7.6 = h0371630_2-python-dateutil = 2.8.1发布= py_0-python编辑器的1.0.4 = = py_0-pytorch = 1.5.0 = py3.7_cuda10.1.243_cudnn7.6.3_0-pytz = 2019.3 = py_0-pyzmq = 18.1.1 = py37he6710b0_0-readline = 7.0 = h7b6447c_5-= 2.22.0 = py37_1请求-= py_0 1.3.0 requests-oauthlib =版本-重试= 1.3.3 = py37_2-rsa = 4.0 = py_0-s3transfer = 0.3.3 = py37_0-scikit-learn = 0.22.1 = py37hd81dba3_0-scipy = 1.4.1 = py37h0b6359f_0-setuptools = 45.2.0 = py37_0-simplejson = 3.17.0 = py37h7b6447c_0-6 = 1.14.0 = py37_0-smmap2 = 2.0.5 = py37_0-sqlite = 3.31.1 = h62c20be_1-sqlparse = 0.3.0 = py_0-statsmodels = 0.11.0 = py37h7b6447c_0它-汇总= 0.8.3 = py37_0-tk = 8.6.8 = hbc83047_0-torchvision = 0.6.0 = py37_cu101-龙卷风= 6.0.3 = py37h7b6447c_3-tqdm = 4.42.1 = py_0-traitlets = 4.3.3 = py37_0-unixodbc = 2.3.7 = h14c3975_0-urllib3 = 1.25.8 = py37_0-wcwidth = 0.1.8 = py_0-websocket-client = 0.56.0 = py37_0-werkzeug = 1.0.0 = py_0-轮= 0.34.2 = py37_0-打包= 1.11.2 = py37h7b6447c_0-xz = 5.2.4 = h14c3975_4-zeromq = 4.3.1 = he6710b0_3-zlib = 1.2.11 = h7b6447c_3-zstd = 1.3.7 = h0b5b093_0-皮普：-astunparse = = 1.6.3-databricks-cli = = 0.11.0-diskcache = = 4.1.0-码头工人= = 4.2.1-大猩猩= = 0.3.0-horovod = = 0.19.1-hyperopt = = 0.2.4.db1-keras-preprocessing = = 1.1.2-mleap = = 0.16.0-mlflow = = 1.8.0-opt-einsum = = 3.2.1之上-petastorm = = 0.9.2-pyarrow = = 0.15.1-pyyaml = = 5.3.1-querystring-parser = = 4-seaborn = = 0.10.0-sparkdl = = 2.1.0-db1-2.2.2 tensorboard = =-tensorboard-plugin-wit = = 1.6.0.post3-tensorflow-estimator = = 2.2.0-tensorflow-gpu = = 2.2.0-termcolor = = 1.1.0-xgboost = = 1.1.1前缀：/砖/ conda / env / databricks-ml-gpu

包含Python模块的Spark包

火花包

Python模块

版本

graphframes

graphframes

0.8.0-db2-spark3.0

R库

R库与R库在Databricks运行时7.0 Beta。

Java和Scala库(Scala 2.12集群)

除了Java和Scala库在Databricks Runtime 7.0, Databricks Runtime 7.0 ML包含以下jar:

组ID

工件ID

版本

com.typesafe.akka

akka-actor_2.12

2.5.23

ml.combust.mleap

mleap-databricks-runtime_2.12

0.17.0-4882dc3

ml.dmlc

xgboost4j-spark_2.12

1.0.0

ml.dmlc

xgboost4j_2.12

1.0.0

org.mlflow

mlflow-client

1.8.0

org.scala-lang.modules

scala-java8-compat_2.12

0.8.0

org.tensorflow

spark-tensorflow-connector_2.12

1.15.0