Cisco Voice Guru

Several months ago, I came across a great script that combines UCCX/IP-IVR with MeetMe functionality.  This allows you to front-end MeetMe calls with a conference number and pin.  Call it a “poor man’s” version of Meeting Place.

Read the rest of this entry »

It’s been asked for.  Here is the latest CCIE Voice Techtorial from Cisco Live! 2010 in Las Vegas.

Right-click this link to download, select “Save As” (PDF)

Most of the content from this troubleshooting post was gathered from the H.323 Gatekeeper Troubleshooting document on Cisco’s docwiki site.  I have tried to summarize the important points below.  Thanks to Zen Beck for pointing me in the right direction.

Troubleshooting H.323 Gatekeeper Registration

To determine registration reject issues, run a “debug h225 asn1” and search for “rejectReason.”  Below is a list of the registration reject reasons and corresponding explanations:

RRJ: rejectReason duplicateAlias

This is usually the result of the gateway registering a duplicate of an E164-ID or H323-ID-another gateway has already been registered to the gatekeeper. If it is a duplicated E164-ID, change the destination pattern configured under a POTS dial-peer associated with an FXS port. If it is a duplicated H323-ID, change the gateway’s H.323 ID under the H.323 VoIP interface.

RRJ: rejectReason terminalExcluded

This is the result of the gateway subnet being disabled in the gatekeeper. Check the gatekeeper configuration. Removing the “no zone subnet gk” command resolves the issue. To remove the command completely, remove the “zone local gk” command.

RRJ: rejectReason securityDenial

This RRJ is the result of the security commands being enabled in the gatekeeper, and the inability of the gateway to match the H.323-ID, E.164-ID, passwords, or security token that the gatekeeper requires. To resolve the issue, check the securiy configuration in the gatekeeper.

RRJ: rejectReason invalidAlias

The RRJ is the result of a no-zone prefix defined in the gatekeeper. Check the configuration on the gatekeeper and add the zone prefix with the proper E.164 address.

Troubleshooting H.323 Gatekeeper Call Routing and Dial Peers

The following is a list of useful show and debug commands used to verify and troubleshoot gatekeeper and gateway call routing issues.

• show gateway - Used to verify E.164 and H.323 alias registration for the gateway
• show gatekeeper endpoints – Used to verify the E.164 and H.323 alias registered with the gatekeeper
• show gatekeeper gw-type-prefix - Used to verify E.164 prefix registrations on the gatekeeper
• show gatekeeper zone prefix | status - Used to verify zone status and configuration parameters
• debug ras - Applicable for gateways and gatekeepers
• debug debug h225 asn1 - Applicable for gateways and gatekeepers
• show dial-peer voice - Used to verify configured technology prefixes under the dial-peers

Troubleshooting H.323 Gatekeeper Bandwidth

When deciding whether there is enough bandwidth to accept a call Admission Request (ARQ), the Cisco gatekeeper calculates the available bandwidth with the following formula:

Available bandwidth = (total allocated bandwidth) – (bandwidth used locally) – (bandwidth used by all alternates).

If the available bandwidth is sufficient for the call, an Admission Confirmation (ACF) is returned, otherwise an Admission Rejection (ARJ) is returned.

Three show commands give you visibility into the bandwidth management on the gatekeeper:

• show gatekeeper zone status
• show gatekeeper zone cluster
• show gatekeeper calls

Troubleshooting Gatekeeper Endpoint Call Admission Issues

Admission Confirmed (Busy Tone Back)

If the Admission Confirmed (ACF) message has been sent by the gatekeeper and arrived at the endpoint side, but the call still received a busy signal, check to see if the terminating IP address in the ACF is an expected valid endpoint IP.  If the ACF has an IP address of the terminating endpoint, remove the gatekeeper and make a direct endpoint-to-endpoint call to see if a call can be established.

Admission Reject (ARJ) rejectReason calledPartyNotRegistered

This is a common reason for rejection. It is captured from the local or originating gatekeeper when the gatekeeper has no information on where the called number should be terminated. There are two ways this problem can occur. One reason is that the call terminates at a gateway, and the gateway is not registered with the E.164 address or with a tech-prefix. To resolve this, make sure the gateway is registered with a tech-prefix to the gatekeeper.

A second possible explanation for this error message arises if the called party is a terminal in a remote zone. It might be that the terminal does not have a proxy enabled in the same gatekeeper zone where it is registered.

Note: Intrazone calls do not require the match of a zone prefix.

ARJ "rejectReason requestDenied"

The rejection shown here comes about because the endpoint-requested bandwidth exceeds the limit configured in the gatekeeper. To resolve this, increase the bandwidth in the gatekeeper using the “bandwidth” command under the gatekeeper mode, or lower the bandwidth request from the endpoint.

Refer to the “debug h225 asn1” for the requested bandwidth in the ARQ (Search for “admissionRequest” in the otuput).  Compare requested value to the “gatekeeper zone status” to see if enough bandwidth is available.

I’m getting off the CCIE Voice bandwagon for a brief moment.  I ran into a real-world scenario today where I needed to pull CDR reports for an end-user’s call activity.  The problem with a generic CDR user report is that it will be truncated if there are more than 100 calls in the output.

The best way around is to do an export of the entire CDR database for the time period in question.  The file will be a .txt file.  Rename it to <filename>.csv and open it in Excel.

The next problem you will encounter is that the dateTimeOrigination column is in epoch time instead of standard time legible by normal human beings.  To convert this, you can use a handy formula that my Twitter-buddy Josh posted on his blog, BlindHog.net.

In Call Manager, the CDR exports are in EPOCH time. Here is the formula to convert epoch time in a Call Manager CDR to a standard format in excel. After using the formula, you will need to format the cell for date and time.

=(((E2-(6*3600))/86400)+25569)

• E2 = cell reference
• 6 = Timezone Offset (this is Central Standard time)
• 3600 = Number of seconds in an hour
• 86400 = Number of seconds in a year
• 25569 = Excel hack because excel counts epoch from 1/1/1900 and most others start at 1/1/1970.

References:

• http://www.mrexcel.com/archive2/44100/51255.htm
• http://www.peterabowen.com/?p=8

External Phone Number Mask on a CME SCCP Phone (Under DN):

ephone-dn 5 octo-line
number 3001 no-reg
name Jacques Strahp
description +3432143001

External Phone Number Mask on a CME SIP Phone (Under Phone):

voice register pool 1
description +3432143001
!
voice register dn 1
number 3001
name Jacques Strahp

SIP Phone Caveats

• You must bind all SIP media and control messages to a particular interface; preferably, the voice VLAN’s default router.  If you do not do this, your SIP phones to sporadically unregister/reregister.  Useful command: “voice service voip” | “sip” | “bind all source-interface Vlan 240” and “registrar server”
• If you do not define an NTP reference for your SIP phones, they will use the time from the last SIP message sender.  Therefore, make sure to enter “voice register global” | “ntp server X.X.X.X”
• Cisco CME SIP phones cannot negotiate codec like the CME SCCP phones.  Therefore, remember to configure “codec g711ulaw” and “dtmf-relay rtp-nte” under each “voice register pool” section.

Intercluster Presence Subscription (CME-to-CME):

Scenario: Extension 5002 at CME HQ, watching Extension 1002 at CME SB.
HQ Configuration:

sip-ua
presence enable

presence
server 10.10.201.1
watcher all
allow subscribe

ephone  2 
blf-speed-dial 1 1002 label "SBPHN2-1002"

dial-peer voice 1002 voip
destination-pattern 1…
session protocol sipv2
session target ipv4:10.10.201.1
incoming called-number 5…
dtmf-relay rtp-nte
no vad

SB Configuration:

presence
server 10.10.200.3
watcher all
allow subscribe
sip-ua 
presence enable

dial-peer voice 5002 voip
destination-pattern 5…
session protocol sipv2
session target ipv4:10.10.200.3
incoming called-number 1…
dtmf-relay rtp-nte
no vad

CUE CLI Command Reference:

• security password expiry days 90
• security pin expiry days 90
• voicemail mailbox owner "SCPHN1" | login pinless subscriber-phone-numbers
• voicemail notification restriction msg-notification

Restrict Inbound International Calls To Single Extension in CME:

This setup is especially useful when the inbound restriction is tied to a particular number.  The gut reaction is to attempt to use CoR, but let’s face it…CoR sucks and doesn’t always work the way you’d expect.

voice translation-rule 2
rule 1 reject // type international plan isdn

voice translation-profile 2-IN
translate calling 2

dial-peer voice 2 pots
call-block translation-profile incoming 2-IN
call-block disconnect-cause incoming call-reject
incoming called-number 1002
direct-inward-dial
port 0/0/0:23

After-Hours Block Patterns in CME:

Remember, the start time starts at the VERY BEGINNING (i.e. 00:00:00 – no secs); the end time ends at the VERY LATEST (i.e. 23:59:59 – max secs).

In Cisco CME 3.4 and later, call-blocking configuration under telephony-service applies to all SCCP, H.323, SIP, and POTS calls that go through the CME router.  All incoming calls to the router, except calls from an exempt phone, are also checked against this after-hours configuration.

after-hours block pattern 1 9011T
after-hours day Sun 12:00 06:59
after-hours day Mon 19:00 06:59
after-hours day Tue 19:00 06:59
after-hours day Wed 19:00 06:59
after-hours day Thu 19:00 06:59
after-hours day Fri 19:00 06:59
after-hours day Sat 13:00 12:00

Basic Gatekeeper Configuration with Default-Tech and Hop-Off

It’s always good to look at gatekeeper configurations, just so you don’t forget!

gatekeeper
zone local US cisco.com 10.10.110.1
zone local SP cisco.com 10.10.110.1
zone prefix US 212* gw-priority 10 HQ-GW
zone prefix US 212* gw-priority 9 SB-GW
zone prefix SP 34*
gw-type-prefix 2# default-technology
gw-type-prefix 617* hopoff US gw ipaddr 10.10.110.2 1720
no shutdown

Basic Gatekeeper CAC for US Zone

gatekeeper
…
bandwidth interzone zone US 32
endpoint resource-threshold
endpoint max-calls h323id HQ-GW 1
no shutdown

Random Notes That Don’t Below Anywhere Else

• One way to limit conference participants is to string ephone-dns together with “no huntstop.”  This way, you can set a restriction lower than 8/16 participants, which is the minimum values possible when working with a G.729r8-enabled hardware conference bridge.
• The command “conference admin” gives users the ability to add/drop/list conference participants in CME.
• Verify CME conferencing using:

show ephone-dn conference
show telephony-service conference hardware

CME-to-CUC SCCP Integration

1. Add call-forward settings for each user’s extension under “ephone-dn”
2. Add voicemail command under “telephony-service”
3. Add voicemail pilot under “ephone-dn”.  Number should be the voicemail pilot.
4. Add DN for mwi under “ephone-dn”.  Use “number 5601 secondaey 5602 no-reg both” | “mwi on-off”
5. Add voicemail port under “ephone”.  Use “vm-device-id” command and make sure it matches what you enter in the CUC port group.

TCP Port Numbers

• TCP 1720 – H.225 Call Setup / FastStart
• TCP 2000 – SCCP
• TCP 2428 – MGCP ISDN PRI Backhaul
• TCP 5060 – SIP Telephone
• TCP 11000-11999 – H.245 Capabilities Exchange / SlowStart

UDP Port Numbers

• UDP 1718 – H.225 RAS Multicast Gatekeeper Discovery
• UDP 1719 – H.225 Gatekeeper RAS Messages
• UDP 2427 – MGCP Control
• UDP 2748 – CTI/JTAPI
• UDP 5060 – SIP Trunk
• UDP 16384 – 32766 (even port numbers) – RTP
• UDP 16385 – 32767 (odd port numbers) – RTCP

CRCX

I’m on a roll tonight.  Here are some more troubleshooting/FYI notes, this time focused on SRST.  The study life of a dedicated CCIE candidate is one constant stream of information absorption occasionally interrupted by family time, eating, sleeping, and relieving oneself in the bathroom.  Sometimes the latter gets delayed by long chapters on gatekeeper behavior (as I’m experiencing right now!).

DLCX

Be aware of the following SRST restrictions:

1. Extension mobility does not work in SRST mode when using call-manager-fallback.  CME as SRST does have extension mobility, but login information is not seamlessly transferred during failure situations.
2. SRST does not have a concept of partitions or calling search spaces. CoR can be used to some extent, but it is a weak-comparison and is not automatically populated on the SRST router.
3. MWIs might be out of synch when recovering from SRST mode.
4. All forwarding (busy, no answer, and forward all) is lost in SRST mode.
5. In SRST mode, all call routing decisions are made on the SRST gateway through dial peer configuration, so anything not explicitly configured on the SRST gateway will not work.

For SRST mode, be sure to configure voicemail under “telephony-service” and to configure an “ephone-dn template” to set the necessary call-forward noan and busy behaviors.

Set your DHCP lease times to several days, or use a local DHCP server to ensure that IP Phones do not reset themselves when their lease expires during a WAN outage.

Use transfer patterns to allow transfers to non-IP Phone destinations.

Survivable endpoints: IP Phones and MGCP Gateways

Nonsurvivable endpoints: H.323 Gateways, SCCP Gateways, CTI/TAPI Endpoints

The rules for call survivability can be summarized as follows:

• If the call involves nonsurvivable endpoints, and a CallManager involved in the call fails, the call fails.
• If the call involves one nonsurvivable endpoint and one survivable endpoint, the call fails only if the CallManager that the nonsurvivable endpoint is registered to fails.
• If the call involves only survivable endpoints, and one or more CallManagers involved in the call fails, the streaming connection between the endpoints is maintained. Note, however, that the endpoints do not have supplementary services available to them after the failure.
• If a hardware-based conference bridge is involved in a call, and the CallManager controlling the conference bridge fails, all calls from nonsurvivable endpoints fail. All calls from survivable endpoints continue in the conference.

  If a software-based conference bridge is involved in a call and the CallManager controlling the conference bridge fails, all calls from the nonsurvivable endpoints fail. All calls from survivable endpoints continue in the conference until the party terminates the call voluntarily. The conference bridge reregisters with an available CallManager.

• If the call involves an MTP or a transcoder, the call fails.
• If the call is in the and a CallManager involved in the call fails, the call fails.

The IP Voice Media Streaming
Application supports only G.711 µ-law and G.711 A-law codecs. It can provide up to 128 streams of conferencing on a standalone server or 24 streams when it coresides with CallManager on the same server.onf

Conference Bridges, Transcoders, and MTPs: Best Practices

• When a media resource is assigned to any MRG, it is no longer available as part of the default MRGL. This means that endpoints that require the use of a media resource in an MRG must have an MRGL configured that contains that MRG.
• Be extremely careful when configuring your codec selections between regions. This is by far the most common cause of transcoder problems.

Troubleshooting MOH

1. Check the MOH server registration status.
2. Check the MRG and MRGL configuration.
3. Verify router configuration for multicast (if multicast is used).
   ip multicast-routing
   ip pim sparse-mode | ip pim dense-mode
4. Verify the multicast capability of the terminating voice gateway.
5. Verify the codec used by all devices involved.
6. Verify the location’s configuration if you are using the centralized call processing model.

H.225/H.245 Summary

H.225 RAS Signaling:

• RAS is the signaling protocol used between gateways and gatekeepers. The RAS channel is opened before any other channel and is independent of the call setup and media transport channels.
• RAS uses User Datagram Protocol UDP 1719 (H.225 RAS messages) and UDP 1718 (multicast gatekeeper discovery).

H.225 Call Control (Setup) Signaling

• H.225 call control signaling is used to setup connections between H.323 endpoints.
• A reliable (TCP) call control channel is created across an IP network on TCP 1720. This port initiates the Q.931 call control messages for the purpose of the connection, maintenance, and disconnection of calls.

H.245 Media Control and Transport

• H.245 handles end-to-end control messages between H.323 entities.
• H.245 procedures establish logical channels for transmission of control channel information.
• It is used to negotiate channel usage and capabilities such as flow control and capabilities exchange messages.

Outside Dial Tone Played at the Wrong Time

• If multiple patterns exist with the same first digit and at least one of the patterns is not configured to provide outside dial tone, outside dial tone is not played unless all the remaining potential matches are configured to provide it.
• Fix 1: Check the box on the 911 route pattern that tells it to play outside dial tone.
• Fix 2: Change the 911 pattern to 9.911 instead and check the box that tells it to play outside
  dial tone (although 9.911 is already included as part of 9.@).
• A good way to isolate the problem is to view the Route Plan Report in Cisco CallManager Administration (Route Plan > Route Plan Report).

Call Forward All (CFA)

• For calls within a cluster, use the Forward Maximum Hop Count service parameter (Service > Service Parameters > select a server > Cisco CallManager) to end a forwarding chain after a specified number of hops
• Use MaxForwardsToDn and StopRoutingOnUserBusyFlag to prevent routing loops that traverse other clusters or legacy TDM equipment.

Called and Calling Party Transformations

1. The External Phone Number Mask on the original calling device (when specified on a route pattern or translation pattern by the Use Calling Party’s External Phone Number Mask checkbox.)
2. A translation pattern
3. A route pattern
4. A route list using route group details
5. Transformation on the terminating device

Methodology for Resolving Call Routing Problems

1. What calling device is having the problem (phone, gateway, CTI ports, Voicemail ports, etc.)?
2. What is the dialed number as dialed by the user or sent by the gateway?
3. What calling search space (and corresponding partitions) are being searched?  Pay attention to the different line/device CSSs that come into play.
4. If a match is made (important word = IF), what pattern is matched?  What partition does that pattern belong to?
5. Is a translation pattern being us?  If so, where is it redirecting the call?
6. Are there any DDIs taking place?
7. If the route pattern goes to a route list (i.e. 99% of patterns), check the route list for digit manipulation being done at that level.
8. If you’re sure the outbound call is hitting the gateway, use my best friends:
   debug isdn q931
   debug voip dialpeer (H.323/SIP only)
   debug voice translations (H.323/SIP only)

Random Notes

• The acronym DDI refers to “digit discard instruction” and not “direct dial in” which is commonly used to refer to direct inward dial (DID) numbers in some countries.

CRCX

Dear fellow CCIE Voice candidates,

I apologize for being out-of-touch these days.  I have a few folders in my Dropbox with massive amounts of checklists and debug explanations.  However, life has been so chaotic in the past few weeks that I have not had the time to faithfully update this blog.

It’s my promise to you that I will add more information to this blog once I pass.  I want this to be a resource that people can use for different Cisco voice questions.  The most important part of the CCIE is the journey that transforms you from a puny n00b with poor troubleshooting skills and no understanding of Cisco’s documentation site to a stone-cold VoIP issue killer.  “One-way audio?  Dropped calls?  That’s nothing!”

If you have been on this journey for a while, suffered long nights, and experienced the chronic aches and pains of caffeine overdose, then you understand what I’m saying.

MDCX

Ok, changing focus…

I am 33 days away from my exam.  I feel pretty confident in my core skills.  I am weak in SIP, however.  Not in the SIP basics like dial-rules, voice register commands, and Early Offer enticements.  Instead, I’m weak in the troubleshooting skills that are required to deploy a basic SIP solution in under eight hours without any major hiccups.  I plan to be reading the old, but not outdated, “Troubleshooting Cisco IP Telephony” book by Cisco Press as well as the “SIP Trunking” book.

SIP is a very cool protocol, though I’m unsure of how or when it will replace the good ol’ ISDN PRI.  This is just one curiosity that has to be tucked away for a rainy day when there are no projects and no CCIE lab looming on the horizon.

I’ve been very tired these days.  In fact, I might not have paced myself adequately because I’m exhausted every evening.  I’m a big believer in good sleep and mental clarity before taking an exam.  I might take a few days easy before diving in for the last stretch.

Good luck to all of you out there!  I’m cheering you on!

I’ll try to post a few more times as the Day draweth nigh.

DLCX

The “Voice (soon to be certified) Guru”