Error Training LGBM Classifier on Databricks

[liuyu930]

HI,
When I tried to train a classification model using LightGBM in a Databricks Serverless environment, I encountered the following error:
['str' object has no attribute 'getParam']

I cannot confirm whether it's an environment issue or a package issue.

However, I am currently attempting to pinpoint the potential environment-related problems.

If anyone has any insights, I would greatly appreciate your guidance.

environment
Databricks Serverless 4
pyspark : 4.0.0

code

`model = LightGBMClassifier(
objective="binary", featuresCol="features", labelCol="buy_flg", isUnbalance=True
)

model = model.fit(trin_date)`

Error

AttributeError: 'str' object has no attribute 'getParam' [Trace ID: 00-678e75eae65d2f9e8bc744eb078227e0-eb6e67874598c4bf-00] File <command-4855832797700963>, line 33 27 # trin_date.display() 29 model = LightGBMClassifier( 30 objective="binary", featuresCol="features", labelCol="buy_flg", isUnbalance=True 31 ) ---> 33 model = model.fit(trin_date) File <command-4855832797700963>, line 33 27 # trin_date.display() 29 model = LightGBMClassifier( 30 objective="binary", featuresCol="features", labelCol="buy_flg", isUnbalance=True 31 ) ---> 33 model = model.fit(trin_date) File /databricks/python_shell/lib/dbruntime/MLWorkloadsInstrumentation/_pyspark.py:30, in _create_patch_function.<locals>.patched_method(self, *args, **kwargs) 28 call_succeeded = False 29 try: ---> 30 result = original_method(self, *args, **kwargs) 31 call_succeeded = True 32 return result File /databricks/python/lib/python3.12/site-packages/pyspark/ml/base.py:203, in Estimator.fit(self, dataset, params) 201 return self.copy(params)._fit(dataset) 202 else: --> 203 return self._fit(dataset) 204 else: 205 raise TypeError( 206 "Params must be either a param map or a list/tuple of param maps, " 207 "but got %s." % type(params) 208 ) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b17e7c41-2947-4628-8639-a4f59e8d5edb/lib/python3.12/site-packages/synapse/ml/lightgbm/LightGBMClassifier.py:2148, in LightGBMClassifier._fit(self, dataset) 2147 def _fit(self, dataset): -> 2148 java_model = self._fit_java(dataset) 2149 return self._create_model(java_model) File /databricks/python/lib/python3.12/site-packages/pyspark/ml/wrapper.py:412, in JavaEstimator._fit_java(self, dataset) 397 """ 398 Fits a Java model to the input dataset. 399 (...) 408 fitted Java model 409 """ 410 assert self._java_obj is not None --> 412 self._transfer_params_to_java() 413 return self._java_obj.fit(dataset._jdf) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b17e7c41-2947-4628-8639-a4f59e8d5edb/lib/python3.12/site-packages/synapse/ml/core/schema/Utils.py:134, in ComplexParamsMixin._transfer_params_to_java(self) 132 self._java_obj.set(pair) 133 if self.hasDefault(param): --> 134 pair = self._make_java_param_pair(param, self._defaultParamMap[param]) 135 pair_defaults.append(pair) 136 if len(pair_defaults) > 0: File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b17e7c41-2947-4628-8639-a4f59e8d5edb/lib/python3.12/site-packages/synapse/ml/core/serialize/java_params_patch.py:122, in _mml_make_java_param_pair(self, param, value) 120 sc = SparkContext._active_spark_context 121 param = self._resolveParam(param) --> 122 java_param = self._java_obj.getParam(param.name) 123 java_value = _mml_py2java(sc, value) 124 return java_param.w(java_value)

[github-actions]

Hey 左眼用来忘记你 (@liuyu930) 👋!
Thank you so much for reporting the issue/feature request 🚨.
Someone from SynapseML Team will be looking to triage this issue soon.
We appreciate your patience.

[levscaut]

Brendan Walsh (@BrendanWalsh) Rana Singh (@ranadeepsingh) could you take a look?

[BastiaanMatt]

I am having the same issues working in databricks, this is my first time using synapse ML and i can't quite pin down what when wrong.

[levscaut]

Probably because you are using pyspark 4.0, try downgrade to pyspark 3.5 which is the latest version we support might help. The support for pyspark 4.0 is on the way by Brendan Walsh (@BrendanWalsh) .