The estimateSizeFactors() function expects a DESeq data object, not a data matrix.

You should do this:

calidata<-read.csv("/Users/pg2597/Desktop/combined/caliisocounts_xyz.csv", header=TRUE, row.names = 1)

mat_data <- data.matrix(calidata)

calibrators <- grepl('cali', rownames(mat_data))

cal <- calibrators

dds <- DESeqDataSetFromMatrix(countData = countData, colData = colData, design = ~ condition)

dds_norm <- estimateSizeFactors(dds, controlGenes=cal)

For further information on this, including what the contents of colData should be and how to set the design parameter, see Quick start and Count matrix input.

Note that people usually use the wrapper function, DESeq(dds), which will do the entire normalisation process for you, including calling estimateSizeFactors(). Your normalised counts could then be accessed with:

counts(dds, normalized=TRUE)

In this case, you are aiming to use control genes though.

Kevin