I am trying to fit a negative binomial regression to count data, with 90 variables. Tried feature selection with GBM, LASSO and random forest. But when I fit the model using glm.nb from MASS, the coefficients are not statistically significant.

Is there a way to specify family = "negative binomial" for gbm or random forest?