State tech agency lacked controls over $158M federally funded broadband project, audit finds

By Charlie Hayward

For MarylandReporter.com

A large coil of fiber optic cable.

A large coil of fiber optic cable.

The Department of Information Technology did not maintain adequate internal controls in managing a $158 million project to upgrade and expand Maryland’s high-speed fiber optics infrastructure that was funded predominantly by federal stimulus dollars, state auditors found.

The auditor found problems with contracting practices, construction oversight, monitoring of sub-grants and cash control. The Legislative Audit report on DoIT also revealed deficiencies in three other areas: major IT development projects; cloud services; and disaster recovery controls.

 One Maryland Broadband Network

The One Maryland Broadband Network is a $158 million engineering and construction project to create a statewide high-speed fiber optics network, supported by $115 million of federal money. The OMBN project will build 1,340 miles of fiber-optic cabling; enhancing broadband service to 1,087 community organizations, such as hospitals, schools, police, other emergency responders, and all levels of government.

A dozen county and municipal officials, including five county executives, two mayor and three county council members joined Gov. Martin O'Malley and Sen. Barbara Mikulski in 2011 for the Inter-County Broadband

A dozen county and municipal officials, including five county executives, two mayors and three county council members, joined Gov. Martin O’Malley and Sen. Barbara Mikulski in 2011 at the ribbon-cutting for the Inter-County Broadband Network in an Elkridge warehouse. .

The project is said to foster economic development, enhance public safety communication, and extend and interconnect existing high-speed networks owned by the state and dozens of county and local governments.

DoIT is ultimately responsible for successfully meeting grant objectives, but major parts of the project are being managed by sub-grantees.

The Inter-County Broadband Network (ICBN) managed approximately 60% of the spending on the project. ICBN is a consortium of nine Central Maryland counties and cities, led by Howard County. (See earlier story.)

The Maryland Broadband Cooperative (MdBC) managed 3% of the funds.

MdBC is a no-profit entity serving Eastern, Southern, and Western Maryland. The state Department of Information Technology is responsible for managing the rest of the work in 15 counties. Project work is predominately being done by contractors, and all work is required to be complete by the end of the month, Aug. 31.

Federal stimulus grant

American recovery and reinvestment act logoThe federal stimulus grant was awarded Sept. 17, 2010, and required a compressed timeline for initial project deliverables. Later that month, DoIT selected a project-management contractor who previously had assisted the state with its grant proposal for the federal money.

The auditor found that DoIT selected the project management contractor from a master list of pre-existing contractors, with no written task order specifying the contractor’s obligations, price, or language protecting the state in the event the grant’s objectives were not fully met.

DoIT disagreed with the auditor’s conclusions on grounds the master contract’s terms were sufficient, even though the grant did not exist when the master contract was signed. The broadband network project was also much-larger in scale than the work anticipated at the time of the award of the master contract.

In response to the audit, the project management contractor issued a letter setting forth its understandings and responsibilities.

The auditor also found that beginning in 2011, DoIT orally agreed to pay the project manager an 8% markup on subcontracted services. However, this arrangement was made without the knowledge of DoIT’s senior management. Further, this markup generally was impermissible in all DoIT contracts. The auditor reported a cumulative markup of $103,000 had been paid to the project manager through July 2012.

Fiber optic cables (By roshan 1286 on Flickr)

Fiber optic cables (By roshan 1286 on Flickr)

Inadequate control over contractors  

DoIT also did not maintain adequate internal control over approval and monitoring of contractors it was managing:

  • Each project segment’s work was stipulated in construction work orders; DoIT neglected to sign the work orders sampled by the auditor, which would bind the parties.
  • DoIT paid contractors but did not maintain “Milestone Acceptance” documentation showing construction work was complete and payments had been earned.
  • Certain DoIT employees had incompatible duties because they selected the contractors to be awarded the work, and subsequently approved their invoices. DoIT disagreed and said it “does not concur that an individual involved in the selection process should be excluded from the review and approval process.”

The auditor also reported DoIT did not have adequate internal control over the portion of the project being managed by the Inter-County Broadband Network, led by the Howard County government. For instance, DoIT:

  • Did not retain evidence that it reviewed ICBN’s invoices to assure conformity with the agreement’s terms before payment.
  • Did not visit construction sites to assure that work was actually being performed.

The auditor reported findings in three other areas.

Major IT development projects and cloud services

DoIT could not satisfy the auditor that it was actively monitoring major IT development projects with significant impact on state operations within executive-branch agencies and commissions. The auditor did not indicate DoIT’s monitoring was deficient, just the documentation that such monitoring was being done was incomplete. This was a repeat finding from the last audit in 2009.

DoIT is managing the state’s migration to cloud collaboration and messaging services. (The “cloud” is somewhat vague term meaning a network of computers using a real or virtual server.)

As of June 2012, almost 900 state workers were utilizing cloud services. The auditor reported several internal control exceptions:

  • DoIT was not assuring that access permissions and file sharing were controlled based on a need-to-know.
  • DoIT has no capability to enforce the prohibition over storing sensitive information (such as proprietary or personal data) in the cloud.
  • Neither DoIT nor the state agencies has the capability to monitor cloud content or exercise control over who accesses such content.

 Disaster recovery

networkMaryland™  is the statewide high-speed network existing before the One Maryland broadband project. The auditor found that DoIT risks a prolonged interruption of major computer operations because:

  • It didn’t have backup configuration stored at a remote site corresponding with four key firewalls. The report indicates DoIT addressed this matter during the audit.
  • It has neither updated nor tested its disaster recovery plan since the major expansion of networkMaryland™ in 2007. The audit report says DoIT will modify its disaster-recovery plan.

Charlie Hayward recently retired after 30 years experience with performance, IT, and financial auditing of a wide variety of government programs and activities. He can be reached at hungrypirana@verizon.net.